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Abstract 



Tlie Florida Comprehensive Program for Career Development: 
K- Universities is a consortium effort to develop a coordinated and 
comprehensive program of career education for students in two 
school districts, a junior college and a university. Components 
of the project include guidance, curriculum, and placement/follow-up. 
This report covers the first year of proie^:: operations. 

Third party evaluation activities by Educational Testing 
Service for the period cov^ered by this report included orientation, 
designing a comprehensive evaluation plan for succeeding years " 
of the project, reviewing project reports and similar sources of 
secondary data, and reviewing analyses of student data based upon 
measures related to project objectives. 

Since much of the data was examined from secondary sources 
or was of an incidental nature, conclusions and recommendations 
are, at this point, very tentative: 

1. The project has initiated activities for all proposed 
components and target groups. 

2. Due primarily to pre-existing structures and programs, 
implementation for the elementary level and for the 
placement /follow- up component across levels is more 
advanced than for other project components and levels. 

3. Exceptional project achievements are noted in the 
areas of communication and inservice training. 



4» Analyses of pre and post student measures of self 

awareness and career knowledge showed no significant 
changes . 

5. To improve continuity, consideration of alternatives 
and allocation of resources, it is recommended that 
more advanced planning of inservice activities be 
considered. The inservice program to be provided by 
the project should be at least tentatively planned 
on an annual basis. 

6. To assure full development and implementation of 
CE programs at the post secondary level during the 
project funding period, a greater proportion of 
project resources should be allocated to this level. 

7. As the project moves from what has been a planning- 
organization phase into more of an implementation 
phase, the project's role with respect to implementation 
should be carefully considered and defined for each 
component and target group level. 

Appendices to the report include the evaluation plan for FY76, 
reviews of self awareness and career knowledge instruments, and a 
review of student data collected during FY75. 
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I. Overview of the Project 



Comprehensive Program for Career Development: K~Universities" 
is a consortium effort involving as principals two public school 
districts (Sarasota County and Manatee County, Florida), a 
community college (Manatee Junior College), and a university 
(University of South Florida). The consortium goal is to 
develop and implement a well-coordinated program of career education 
for all student groups served by the member institutions. 
Through central planning, coordination, and communication, the 
project is attempting to reduce duplications of effort and problems 
associated v/ith articulation of students between member institutions. 

The structure of the project includes three components' (Guidance, 
Curriculum, and Placement/Follow-up), each functioning at four 
levels (Elementary, Secondary, Junior College, and University). 
Lessor, but similar, services have been extended to private 
schools, vo~tech schools, and community groups. Project activities 
are coordinated by a central management staff of six professionals ' 
in cooperation wiUr a wide variety of advisory and task force 
committees composed of staff members from the consortium institutions. 
State direction for the project is through the Bureau of Vocational 
Research and Evaluation, Division of Vocational, Technical, and 
Adult Education. 

The consortium project became operational on July 1, 1974, after 
an initial planning phase of 90 days, and io completing the first 

f 
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full year of operation. The annual budget for the project is 
approximately $180,000. 
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II. Overview of the Evaluation 



In February, 1975, Educational Testing Service entered into an 
agreement with the Florida Department of Education, Division of 
Vocational, Technical, and Adult Education to plan and conduct an 
evaluation of the Florida Comprehensive Program of Vocational 
Education for Career Development as required by the Rules and 
Regulations of Section 142(c) of Part D, Vocational Educational 
Amendrr^^nts of 1968, P. L. 90-576. 

T^v^o limitations were considered in structuring evaluation 
activities for the remainder of FY 75: First, the Career 
Education Consortium Project had been in operation for six 
months. Therefore, since ETS had had no previous association 
with the project, considerable time for orientation and planning 
would be required before implementation of a meaningful evaluation 
design would be possible. Secondly, with only six months remaining 
in the fiscal year, the probability of detecting program effects 
through change measures was remote. 

The approach of ETS to the evaluation v/as to use the remaining 
months of FY 75 primarily to become familiar with the structure 
and philosophy of the Career Education Consortium project and 
to design a comprehensive evaluation plan to be implemented in 
FY 76. In addition, ETS sought to collect enough process data 
from secondary sources to determine the actual status of the 
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project, and to review available analyses of student assessment 
data for evidence of possible project effects. 

More specifically, the goals of the evaluation were as follows: 

1. To develop, define, and establish working relationships 
with the various groups of people who have potential 
for input of information into the evaluation. 

2. To organize existing information about both process 

and product in the schools and at the college, including 
the "before" measurement data that might be available 
at one or more schools, 

3. To assemble information to support a certification 
that the program was, in fact, in operation. 

4. To identify processes and activities which were not 
on schedule in their development and thus endangered 
the success of the program. 

5. To solidify sampling and data collection procedures. 

6. To develop the final evaluation design in congruence 
with the discovered realities of the program. 

7. To process such before-after information about students 
in the first year of the program as was deemed accurate 
and relevant. 

Me thodology 

Evaluative judgments of the status and achievements of the Florida 
Career Education Consortium in this report are based upon 
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information from three sources. First, project documents and 
products such as quarterly reports, newsletters, and meeting 
minutes were reviewed. These documents have, at this point, 
been accepted as accurate and representative without extensive 
auditing. The evaluation design for succeeding years will, 
however, include a system of checks and balances to verify such 
project reports. 

Secondly, the evaluators made frequent visits to the project 
site to discuss with project management goals, problems, and 
achievements to date. Although, the primary purpose of these 
visits was to gather information for planning an evaluation 
design, each visit provided incidental information relative to 
program status. 

A third source of evaluation information was test data collected 
by the two school districts according to a design initiated 
prior to ETS involvement. A pre-post control group design was 
used. Measures of self -awareness and career knowledge were 
administered in grades 1-9. Further information on sample 
sizes, instruments used, and findings for this study is reported 
in Appendix A of this report. 

Limitations of the Study 

The most obvious and serious limitation of the present evaluation 
of '^A Comprehensive Program for Career Development: K-Universities" 
is the lack of systematically collected objective data. Due to 
constraints of time and developmental sequence, ETS had to depend 
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upon data from secondary sources such as project: reports and ^ 
student achievement measures collected by the district- Such 
data is extremely susceptible to misinterpretation since the 
evaluator may be unav/are of the procedures and politics involved 
in the data collection or reporting, or may not have data on 
some critical alternate hypotheses. The limited contact of ETS 
with the project also makes it highly probable that there have 
been significant project activities and accomplishments of which 
the evaluator is simply unaware. Consequently statements in 

this report that "No evidence of was found'* 

should not be interpreted as saying there has been no project 
activity in a given area. The data upon which this report is 
based is largely incidental. No systematic search for evidence 
of any particular activity or outcome was conducted. 
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III. Major Evaluarlon Activities 



O rientation and Planning 

The emphasis of the evaluation activities for FY 75 was on 
orientation of the evaluators and on development of a compre- 
hensive evaluation plan to be implemented in FY 76. 

A, Orientation activities included developing working relation- 
ships with the several levels of project management as well as a 
careful review vof project planning documents and progress reports* 
Table 1 lists meetings between ETS staff and the Career Education 
Consortium (CEC) management. Table 2 lists those project 
documents received and reviewed by the evaluator* Linking the 
meetings of the' evaluation and management staffs, many phone 
calls and written communications were exchanged. Each meeting 
or exchange brought a greater understanding of project philosophy, 
goals, and activities, and aided the development of an evaluation 
strategy responsive to project needs and valid as an assessment 
of project accomplishment. 

B. To aid in planning data collection for subsequent periods of 
the projects, instruments used in the FY 75 evaluation were 
carefully reviewed for technical quality and relationship to 
project objectives. These reviews are included as Appendix B. 
Additional instruments V7ere also collected and reviewed for 
appropriateness and these are also listed in Appendix B. 

lo 
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Table 1 

Site and Project Visits 
for the Evaluation of the Florida Comprehensive Program 
for Career Development : K-Universities 
for 1975 



City Date Purpose Persons Contacted 



Tallahassee 



Dallas, 
Texas 



Bradenton 



Bradenton 



Bradenton 



Bradenton 



Tallahassee 



Bradenton 



1/9 Review proposal 



3/28-3/30 National Conference 
for 3rd Party 
Evaluators 

3/18-3/21 Orientation 



4/15-4/15 Review Evaluation 
Plan 

4/28-4/29 Present Draft 

Evaluation Plan 

5/3 Attend workshop for 

faculty and 
department leaders in 
Consortium institutions 

6/19 Review previous 

evaluations conducted 
in Florida 

5/25-5/27 Review test schedule 

plans for 1975-76 with 
local districts 



Dr. Bert, Dr. Selman, 
Dr. Tully 

Dr. ^Bert, CEC staff, 
USOE staff. Dr. Lyles 



CEC staff., Mr. Bellum, 
Dr. Freijo, Dr. Bur ley, 
Mr, Bucklin, Mr. Laudano, 
Mrs. Swatzell, Mrs. 
Foerster, Dr. Pelletti, 
Advisory Committee 

Dr. Bert, Dr. Selman 



CEC staff 



Dr. Selman, Dr. G. Simons, 
Workshop participants 



Dr. Bert 



CEC staff. Dr. Bert, 

Mr. Laudano, Mr. Bucklin, 

Dr» Pelletti 
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Table 2 

Documents Reviewed 
for the Evaluation of the Florida Comprehensive Program 
for Career Development: K-Universities 
for 1975 



Bureau of Occupational and Adult Education, U. S. Office of Education, 

Department of Health, Education, and Welfare. Handbook for the evaluation 
of Career Education Programs (Draft) , Washington , 19 74 • 

Dale, J. School Leaver Statistics for 1973-74 , Career Education Consortium. 

Erie jo, T. D. Sarasota County Career Education Evaluation Report No, 2 : 
Elementary Students^ Pre-test Results . 

Holmb raker, H., et, al. New Directions Program Annual Report 1973-74 , 
Follow-up Study . Sarasota County Vocational-Technical Center, 1974, 

Martin, G, S. Planning for Comprehensive Program of Vocational Education 
for Career Development . Career Education Consortium, Final Report, 

Pillott, G, M. and Ehlers, R. W. District Plan for Career Education , 
Sarasota, 1974. 

Preston, J. R. State of Florida Evaluation Committee on the School-based 
Job Placement Service Pro;iect, An Interim Report . Florida State 
Department of Education, 1975. 

Selman, J. W. A Comprehensive Program of Vocational Education for Career 
Development K-University , First Quarterly Report (July 1-September 31 , 
1974). 

Selman, J. W. A Comprehensive Program of Vocational Education for Career 

Development K-University , Second Quarterly Report (October 1-December 31 , 
1974 ), 

Selman, J. W. A Comprehensive Program of Vocational Education for Career 
Development K-University , Third Quarterly Report . Bradenton, Florida, 
Manatee Junior College, 1975. 

Selman, J. W. A Comprehensive Program of Vocational Education for Career 
Development K-University i> First Interim Report . Bradenton, Florida, 
Manatee Junior College, 1975. 

Woolley, W. Florida View Vital Information for Education and Wor k. 

Chipley, Florida (Florida View Center at the Panhandle Area Educational 
Cooperative.) 
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Documents Reviewed (Continued) 



Periodicals ; 

The Articulator ^ Newsletter of the Career Education Consortium of the 
State of Florida, Volume II, Number 4, April 1975. 

The Articulator , Volume II, Number 5, 1975. 

Community Career Line Quarterly Newsletter on Career Education , Volume 1, 
Number X, Summer, 1975. Career Education Consortium, Manatee Junior 
College, Bradenton, Florida 33507. 

OTHER 

Employment Forms : 

"Adult Office Occupations Training Program Follow-up Survey" 
Articulation Problems, Admission Policies, Procedure and Requirements 
"Career Education How Did You Do It?" 
Manatee Junior College Follow-up Survey Form 
"School Board of Manatee County Follow-up Survey" 

"The School Board of Sarasota County Office of Placement and Follow-up Survey" 
Abstracts : 

Future Life Analysis and Guidance Program Proposal for Sarasota County, Florida. 

"The Infusion of Career Education Competencies in the Pre-Service Preparation 
of Educational Personnel." Abst-ract, College of Education, University 
of South Florida, Tampa. 

Proposals : 

Buchlin, Walter P. Career Education Program 13.554 Proposal . Manatee 
County, Bradenton, Florida, 1975. 

Federal Guideline Objectives for a Comprehensive Program of Vocational 
Education for Career Development K-University . 
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C. Several drafts related to the establishment of evaluation 
designs for both products and process were submitted and 
discussed; These led to an improved understanding of project 
structure and goals and contributed to the final evaluation 
design to be implemented in the second year of the project. 

D. The principal product of orientation and planning activities 
was a comprehensive evaluation plan based upon project structure 
and goals and designed to provide objective data from a variety 
of sources for the future assessment of project achievements. 
This plan is briefly described in Appendix C» 

P rocess Evalua tion 

Although it was not possible to implement any systematic design 
for process evaluation in the short time available in FY 7A-5, 
several evaluation activities provided incidental evidences of 
the status of program components. 

A. Each site visit listed in Table 1 provided information on 
current and past project activities through staff interviews, 
participant interviews, and direct observation. 

B. A thorough review of the project proposal and related 
planning documents and quarterly progress reports provided a 
comparison of planned activities and schedules with actual 
occurrences. 

C. A search of project quarterly reports for activities 
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representative of each management function at each target group 
level was conducted to provide evidence of the developmental 
stage of programs for each target group level. Table 3 is 
based upon this search. 

Product Evaluation 

Product evaluation, in this context, refers to the analysis of 
student changes as measured by a variety of cognitive and affective 
measures. Prior to ETS*s involvement with the Career Education 
Consortium project, two school districts in the CEC had contracted 
for the collection and analysis of data related to the student 
objectives of the project. ETS agreed to review the analyses 
of these data and to comment on possible implications for future 
project activities. A report of this activity is included as 
Appendix A. No additional data related to product evaluation 
was collected by ETS. 
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iy« Findings 

Findings related to the CEC project are discussed in two sub- 
divisions: program status and program accomplishments. All 
findings are based upon relatively limited information since no 
systematic design for information gathering was implemented. 
Judgments are based primarily on interactions with the project 
management staff and examination of project records and products. 

Program Status 

Every new project will go through a period of definition in which 
the roles and goals of the projects are continuously changed and 
refined. Such a period was very evident in the CEC project. Even 
though there was a formal planning effort before implementing the 
project, alterations in structures, roles, and goals evolved in 
response to the needs of funding agencies, the consortium member- 
ship, and the philosophies, skills, and personalities of the 
staff involved. Much of the confusion associated with initial 
definition has passed. The CEC management staff seem to be 
confident of their role in promoting articulation and career education 
in the service area. However, with many institutions and agencies 
each having a vested interest in the project, pressures to shift 
project emphases and/or resources will always be present. In such 
situations, it is extremely important that project plans be explicitly 
stated and communicated to all groups involved. 

13 
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One indication that perhaps the major battles associated with 
project definition have passed is that the project has begun to 
systematically develop programs and services. Some CEC programs 
are in the very early stages of development; others are very near 
operational status. The more fully developed programs are, in 
general, those that could build upon substantial previous efforts. 
This would include Placement and Follow--up which was partially in 
operation before the Consortium was formed and curriculum efforts 
at the elementary level where district staffs were already involved. 

Although there is ' considerable variance in the level of development, 
development of programs for all project defined target groups has 
begun. A systematic review of the four quarterly reports from the 
project to the Department of Education produced examples of project 
activities for almost all levels and all management functions. 
Some of these exemplary activities are included in Table 3. The 
only area in which no evidence of activity was found was evaluation 
at postsecondary levels and for private schools and community 
groups. This is perhaps indicative of the developmental status 
of programs in these areas. 

Management functions in a developmental effort proceed in sequence 
from planning to implementation, evaluation, and dissemination. The 
absence of evaluation activities at the postsecondary level is 
evidence that the programs for this level have not yet been sufficiently 
implemented to justify evaluation. 

20 
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Table 3 






Exemplary Project Activities 
Organized by Target Group 
and Management Function 




Planning 


Implementation 


Elementary 
School 


Discussion of student input 
through rap sessions with 
students (2nd qtr. , p. 46) 


Elementary School Orienta- 
tion-Es tablishment- 
Operations-Assis tance 
Packets (2nd qtr,, p, 46) 

Consult with schools in plan- 
ning special "Career Week" 
activities (3rd qtr., p. 101, 
1st qtr. , p, 121) 


Secondary 
Schools 


Plans for Career night, 
day, and week programs 
(2nd-qtr., pp. 167-68) 

Plans for forming a student 
advisory committee (2nd 
qtr. , p. 46) 


In-service training programs 
on curriculum improvement, 
need for career education on 
secondary level, (2nd qtr, , 
pp. 5, 94, 95, 96, 1st qtr. , 
p. 121) 


Manatee 

Junior 

College 


Set up communications be- 
tween CEC and MJC facul- 
ty, staff, and student 
body. 

Setting up of Coordina- 
ting Committee (2nd 
qtr. , p. 54) 


Set up student advisory com- - 
mittee (2nd qtr., p, 58) 

Established Career Educatio-i 
Task Forces at MJC (3rd qtr, ,- 
pp. Xy 124, 1st int., p, 5) 



Interfacing of the consortium 
staff with MJC staff to identi- 
fy existing career education 
activities (3r<J qtr. , p, 9) 



University of 
South Florida 


Formation of postsecond- 
ary coordinating commit- 
tee with USF (2nd qtr., 
p. 53, p. 2) 


Establish career education 
task force at USF 
(3rd qtr. , p, 1) 


Other 

■21 


Survey of private schools 
to identify areas of 
interest (2nd qtr,, p. 41) 

Request for clarification 
of individual school dis- 
trict involvement with 
private schools 
(2nd qtr, , p. 46) 


Dr. Mary Green appointed to 
coordinate efforts for 
private schools (2nd qtr. , 
p.. 41) 

In-service training sessions 
for postsecondary , secondary, 
and private schools 
(1st int. , p, 258) 
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Table 3 (Continued) 



Evaluation 



Dissemination 



Workshop Evaluations 
(1st int. , pp. 209-239) 

Consultant Evaluation 
(1st int. , pp. 240-252) 



Workshop with Manatee Elementary 
Schools and Sarasota Elementary 
Schools (1st int., pp. 209, 210 
212, 213, 214, 215,...) 



Evaluation of Job Place- Presentations at several high 

ment Services schools (workshops, etc.) 

(1st int., p. 284) (2nd qtr. , pp. 5, 94, 95, 96) 



Staff members lectured on 
Career Education at MJC 
(3rd qtr. , p. 15) 



Staff members lectured at USE 
on Career Education 
(3rd qtr. , p. 15) 



In-service training session 
for postsecondary and secondary 
and private schools 
(1st int. , p. 258) 
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Due to previous State funding for career education in the districtis, 
programs at the elementary and secondary levels are further along 
in their development than programs at the postsecondary level* 
Consequently, programs at the postsecondary level are still in a 
planning-implementing phase and evaluation is not yet appropriate. 

Prior to implementation of the Consortium, a structure for the 
implementation of career education and a staff with full-time 
responsibility for career education existed in the two member school 
districts. No such structure previously existed at the post- 
secondary level or in private schools and community groups. It 
is logical to expect programs in these areas to devlop more slowly 
since the consortium must "start from scratch". 

Based upon admittedly limited information, the evaluator can identify 
no component or target groups for which the level of project 
activity during the first year was not within realistic expectations. 
As Table 3 indicates, there has been some project planning and 
implementation activity for every target group level. The level of 
project activity and resource commitment at each level is related to the 
status of existing programs at the time the consortium v/as initiated. 
Efforts at the postsecondary level have largely been directed toward 
initial communications, orientation, and planning; programs for 
the school districts have built upon previously existing operational 
programs through coordination of district efforts and extensions 
of in-service training. ^ ^ 
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ProRram Accomplishments 

As previously noted, project activities in all areas meet normal 
expectations for a first year developmental effort. However, in 
two areas accomplishments of the project are exceptionally notable. 
Perhaps the most visible program accomplishments are in the areas 
of communication and articulation. Ten committees and task forces 
involving some 50 different persons have been organized by the 
project and meet on a regular or periodic basis. A general project 
newsletter, The Arti culator, is distributed monthly. A placement 
and follow-up newsletter has also been distributed and a third 
effort. Community Care er Line, is under development. The project 
staff also averages over 200 contacts per month with the staffs 
of the Consortium institutions. This intergroup communication 
and participation is essential to a cooperative effort and the 
project should be commended for accomplishments in this area. 
Several problems or potential problems have been eliminated by 
bringing the right persons or groups together for discussions. 
The project has also been extremely active in disseminating project 
information to outside groups through conferences, direct mailings, 
and news releases. 

Another area in which substantial activity is evident is in-service 
training. A number of workshops for both administrators and 
faculty have been conducted by the project. Six major workshops 
are reported in addition to workshops conducted by the two district 
CE staffs. Again, there has been no opportunity to systematically 
assess the effectiveness of these activities, but the one session 
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attended by an ETS evaluator was marked by good participation and 
favorable responses from participants and evaluations compiled by 
the CEC project staff indicate that participants judged the work- 
shops to be beneficial and well conducted. Summaries of questionnaire 
data collected from participants of several workshops are included 
in the project's interim report for FY 75. 

The findings and accomplishments noted in this report are not 
necessarily representative of the most significant accomplishments 
of the project. They are simply those most evident in the relatively 
limited contact between evaluator and project. 
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V, Problems Encountered in the Evaluation 

In evaluating any developmental effort, some problems come with 
the job. In the CEC project, there was considerable apprehension 
about the role of a third party evaluator. The two districts 
involved had previously contracted with another third party and 
were concerned that additional data and interference with classes 
would be required. Of course, those directly involved in the 
project were anxious for it to succeed and, to some extent, viewed 
the evaluator as a threat to project success and to project funding. 

Much of this initial apprehension has been relieved through co- 
operative planning and frequent face-to-face communication betx^een 
the ^evaluator and the project. It was also agreed that ETS 
would not collect any additional data in the two consortium 
districts, but would review and interpret data collected by 
district's third .party evaluator. 

In attempting to design an evaluation plan for FY 76, several drafts 
were submitted by ETS and numerous communications were exchanged 
among the evaluator, the State project director, and the project 
staff before approval was obtained. This delayed other evaluation 
activities for FY 75, but should result in an improved design for 
subsequent years. 

In attempting to interpret data on student objectives, the evaluator 

was handicapped by a lack of supporting process data for the Sarasota (CO 

and Manatee school districts. Specifically, there was no data to 
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verify an actual treatment difference existed in designated "treatment" 
and "control" classes.' Consequently, in measurement areas where no 
student differences were found between treatment and control groups, 
the alternate hypothesis, that no treat*».ent differences existed, 
remains plausible. 

The role of the third party evaluator, as presently defined, is 
a difficult one. He must simultaneously consider the information 
needs and concerns of USOE, the State DOE, and the local project 
management. Considerable effort has been required to establish 
working relationships that will allow this role and further 
cooperation will he needed. 

Dissemination Activities 

No evaluation data or findings related to the CEC project have 
been disseminated by ETS. The evaluation plan for FY 76 has, how- 
ever, been shared with other third party evaluators working on 
similar projects. 
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VI. Conclusions and Recommendations 

Judgment concerning project status, project effects and prognosis 
of the project* s future must, at this point, be very tentative. 
The project is in its first year; the evaluation began six months 
into that year. 

In general, the evidence is sparse. Conclusions must be considered 
in this light. 

Planning 

Much of the activity of the CEC project during the first year has 
been in the areas of organization and planning. Based upon admittedly 
superficial observations, it is the evaluator's opinion that the 
project has been very successful in this area. It is never ev^sy to 
coordinate the activities of several groups to maximize benefits 
to all. Perhaps the key to successful group endeavors includes 
involvement and communication. There is substantial evidence that 
the CEC project has established both a high level of involvement 
and almost continual communication among the staffs of the consortium 
institutions. The v/ide variety of active planning committees, 
advisory committees, and task forces have provided a forum for input 
and discussion of e^ach major project decision and activity. These 
groups, combined with frequent ^visits by the project staff, formal 
newsletter communications, and joint in-service training workshops 
are commendable as raeans of keeping project participants involved 
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and well informed. Open coimnunication lines can help a project 
avoid many problems and quickly solve many others. The" structure 
and activities of the Florida Career Education Consortium have 
served well the functions of planning and communication. Some 
minor problems of role and personality conflicts have been identified, 
but progress is evident in solving these and they do not, at this 
time, pose any threat to project goals. 

Imp 1 eme n t a t i on 

In accord with developmental sequence, more project activities and 
resources will be devoted to the management function of implementation 
during the second operational year# It is now important for the CEC 
project to more clearly define its role in implementing career 
education programs. A wide range of roles are represented by present 
activities. At the district level, the CEC role has been one of 
consultant assistance, in-service training, and coordination of group 
activities. The central staff of the CEC project has not been in- 
volved to any extent in the day-to-day delivery of services directly 
to students. On the other hand, the CEC central staff has accepted 
responsibility for almost all phases of the implementation of 
placement and follow-up procedures at MJC. It is not clear in 
project documents which of these roles represents the "model" 
and which represents a temporary response to existing conditions. 
For smooth implementation of additional career education elements, 
it is important that the Consortium consider what its role should 
be and conunuuicate this clearly to all Involved. Consideration should 
also be given to what the structure and role of the Consortium will 
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be v/hen present developmental funding is exhausted. 
In-service Training 

There V7as considerable activity in the area of in-service training 
during the first year of the project and it 2.s assumed that this 
v/ill continue during the second year *at a similar level, \fliile 
the workshops conducted to date have each been beneficial to the 
project in their own right, they have not been parts of any defined 
sequence based upon needs and project priorities. Workshops were 
independently conceived and planned. The utilization of project 
resources for in-service training might be improved if more were 
done in the area of long range planning of in^-service needs. 
Planning in-service programs at least a full year at a time would ^ 
give the project an opportunity to weigh alternative workshops in 
terms of available resources and long range goals. 

Internal Evaluation 

Although the CEC project staff admittedly have little experience 
and expertise in evaluation, their efforts in this area are commendable. 
The project manageirient is sensitive to the need for evaluation and 
continually seeks and utilizes feedback from a variety of sources. 
VThile project plans designate one person as having the major re- 
sponsibility for internal evaluation, in reality, all of the central 
project staff have been involved in the design and implementation 
of evaluation activities related to their component and target 
group responsibilities.. Project reports have included participant 
evaluations of workshops and newsletters. State-of-the-art and 
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problem surveys have also been conducted. 

Since the third party evaluation cannot provide adequate feedback 
to the project on each project activity or even each component- 
target group combination, it is essential that the CEC staff continue 
to carefully examine the effects of their activitiee^ through formal 
and informal evaluation strategies. As familiarity with the project 
increases and as informational needs arise, ETS may suggest specific 
studies or activities in the area of internal evaluation, but at 
the present time the level of project activity in this area appears 
adequate. 

In examining the CEC project by target group level (elementary, 
secondary, pos tsecondary , etc.)?. it is evident that there has been 
an attempt to provide services to all on a relatively equal basis. 
\^^hile this strategy will serve to keep everyone happy, it is perhaps 
not the best utilization of project resources if, in fact, the 
project *s goal is to establish a balanced and coordinated career 
education program at all levels. In establishing project priorities, 
both the status of present programs at various levels and available 
resources at various levels must be considered. In both consider- 
ations, the elementary and secondary level programs of the CEC have 
an advantage over postsecondary programs. The districts involved 
have full-time career education staffs not available in the post- 
secondary institutions. The districts have also been involved in 
a conscious effort to develop CE programs for a longer period of 
time. 
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To help postsecondary programs "catch up" to the developmental 
level of elementary and secondary programs, a larger share of CEC 
resources should be allocated to this target group level. This is 
not meant to imply that elementary and secondary programs a.re fully 
developed. They, too, have a long way to go. However, the CEC role 
at the elementary and secondary level can be one of coordination 
and consultant assistance with implementation more the responsibility 
of district CE staffs. 

Programs in career education in private schools, in vocational- 
technical schools, and in other non-consortium institutions are also 
in a fledgling state. Within present Consortium resource limitations 
it is not realistic to expect fully developed programs in these 
institutions. However, the Consortium may serve as a catalyst through 
communications and joint in-service training to encourage these 
institutions to develop their o\m programs. 

Project plans for the second year of operation imply a shift in 
priorities to a pattern congruent with the one suggested here. Future 
evaluations will examine the* extent of changes in activities and 
subsequent effects. 
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In t r o du c t io n 

The purpose of this report is to describe and review the procedures 
and conclusions of two evaluation studies of career education conducted 
in Florida. During 1974-75 the University of South Florida (USF) 
conducted evaluations of the Career Education Projects in Manatee and 
Sarasota counties. Various reports were prepared by USF which focused 
on student data collected relative to the objectives of the Florida 
Comprehensive Program for Career Development. The descriptions, 
interpretations and conclusions presented in this report are based 
on information included in those USF reports for Manatee and Sarasota. 

In addition to providing an interpretive perspecti^ve for the specific 
results and conclusions included in the University "of South Fl.orida 
reports, the present report will provide additional interpretations 
and conclusions regarding student responses and performance. When- 
ever possible cross-county conclusions are also included. 

The reader can expect to find sections in this report devoted to 
discussions of a) student sampleSj b) procedures and instrumentation, 
c) variables and analyses, and d) results and conclusions. Sample 
sizes and sample composition, as well as the sample selection process 
are examined in the section on student samples. Brief descriptions 
of each instrument used for collecting student data are included in 
the procedures and instrumentation section. Independent and dependent 
variables' included in the USF analyses are described in the section \) 
on variables and analyses. The student description variables used 
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by USF in presenting group performance data are also identified in 
that section. The results and conclusions section includes a) a 
presentation and discussioji of the USF results and conclusions, and 
b) additional interpretations and conclusions based on group performance 
data presented in the USF reports. 

The following list identifies the USF reports examined in the 
preparation of this report: 

Manatee County Career Education Evaluation Report No, 1: 

Middle School Students' Pre-Test Results 
Manatee County Career Education Evaluation Report No. 2: 

Elementary Students' Pre-Test Results 
Manatee County Career Education Evaluation — Final 

Report 

Sarasota County Career Education Evaluation Report No 1: 

Junior High Students V Pre-Test Results 
Sarasota County Career Education Evaluation Report No. 2: 

Elementary Students' Pre-Test Results 
Sarasota County Career Education Evaluation Report No. 3: 

Pineview Pre-Test Results 
Sarasota County Career Education Evaluation — Final 

Report 

r< ; 
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Description of the Student Samples 

A description of the sample and sample selection process provides 
important information for interpreting and generalizing results. 
This section attempts to provide information about the samples and 
sample selection process involved in the USF evaluation studies 
conducted in Manatee and Sarasota counties. 

Sample size and sample composition data were examined in an effort 
to answer the following types of questions: 

a) Were the samples large enough to allow stable results? 

b) Was the loss of students because of incomplete data, or 
for other reasons such that one should be concerned 
about the representativeness of the sample? 

c) Is there any reason to believe that the Manatee students 
and the Sarasota students cannot be considered samples 
from the same population? 

The discussion of sample sizes will include information regarding 
the total sample sizes for each county as well as the sample 
sizes for each grade level. Sample composition data are described 
for the following independent variables used by USF in conducting 
analyses of covariance : sex^. race, and occupation of the head of 
household. 
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Sample Selection and Sample Sizes 



Manatee County student sanples were selected from a) three 
experimental and three control elementary schools, and b) three 
fsxperimental and -three control middle schools. In all cases, a 
ten percent random sample was selected using students in each 
grade in each school as individual sampling frames. This 
means, for example, that ten percent of the students attending 
grade 1 in school A were randomly selected for the evaluation 
study. For each experimental and control elementary school, 
students were sampled at each of grades K, 1, 2, 3, 4, and 5. 
The reports specify that a total of 289 students were selected 
initially, and for 272 of those students, pre-post test data 
were obtained. 

The middle school samples included students from grades 6, 7, 
and 8 in each of three experimental schools. The samples from 
the control middle schools involved sampling grade 6 from one 
school, grade 7 from another school, and grade 8 from still 
another school. The evaluation reports indicate that a total 
of 224 students were selected initially, and for 109 of those 
students, pre-post test data were obtained. This loss of 115 
students is rather substantial, but at least 71 of those students 
were sixth grade students. 

Sixth grade students in the middle schools were pre- tested with 
the same instruments that were used for seventh and eighth grade 
students. The experience of administering the pre-tests led to 
concluding that the tests were not appropriate for six graders, 
and therefore, six graders were excluded in the post-test data 
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Sarasota County student samples were selected from a) three 
experimental and three control elementary schools, b) one 
experimental and one control junior high school, and c) grades 
4 through 9 at Pineview. For the elementary and junior high 
school samples, a ten percent random sample was selected using 
students in each grade in each school as sampling frames • For 
each experimental and control elementary school, students were 
sampled at each of grades K through 6. The reports specify that" 
a total of 455 students were selected initially, and for 406 of 
those students, pre-post test data were obtained. For each 
experimental . and control junior high school, students were 
sampled at each of grades 7, 8, and 9. The evaluation reports 
indicate that a total of 211 junior high school students were 
selected initially, and for 169 of those students, pre-post 
test data were obtained. 

Since Pineview is a school for gifted students, the design 
for data collection and analysis was different from that involved 
in the study for other school samples. The design was one 
involving pre-post comparisons only rather than comparison to a 
control group. A ten percent random sample of all students 
was selected, yielding a sample of 29 students in grades 4 
through 9. For all 29 students, partial or complete pre and 
post-test data were obtained. 



Based on inspection of the tables presented in the USF reports, 
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several points seem appropriate for mention. Although the 
original "Manatee middle school sample size reported was 224, 
Table 1 of the pre-test results report' shows partial or complete 
pre-test data on only 217 students* Similar data are evident from 
Table 1 of the junior high school pre-test results report from 
Sarasota County* Although the original Sarasota junior high school 
sample size reported was 211, apparently partial or complete pre- 
test data were obtained on only 205 students. 

A rather curious discrepancy in sample sizes appears when one 
inspects Table 1 of the Manatee elementary school pre-test 
results report* That particular table indicates that partial 
or complete pre-test data were obtained for 328 students* This 
number is larger than the original sample size of 289 reported 
elsewhere in that report* This curious phenomena also appeared 
in the Sarasota elementary school reports. Table 1 of the 
Sarasota elementary school pre-test results reports indicates that 
partial or complete pre-test data were obtained for 485 students* 
This number is larger than the original sample size of 455 
reported elsewhere in that report. 

The discrepancies, when one compares original sample sizes with 
the size of samples for which both pre and post-data were available, 
indicate some loss of students from the sample. Some loss could 
be expected as a result of students leaving school or transferring, 
etc. Discrepancies also existed when the original sample sizes 
were compared against the sample sizes on which pre-test data 
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were reported. These discrepancies for the elementary school samples 
in Manatee and Sarasota are quite disturbing and most unfortunate. 
With respect to the discrepencies between pre and post-testing, this 
writer would like to view the loss of students as being a "random** 
loss that did not lead to a biased sample. The USF reports do not 
include explanations of the losses. 

No data were presented to indicate the sample sizes for experimental 
and control groups separately, for either the Manatee or Sarasota 
county study. Because of the sampling procedures used in Manatee, 
however, it is assumed that the experimental students out-numbered 
the control students. 

Grade level sample sizes, as such, did not appear in the USF reports. 
The USF reports did include tables of means, however, and these 
tables indicated the number of students in each grade level for 
which post- test data were available. One may assume that students, 
for whom post-test data were reported, would not have been tested 
in the spring of 1975 unless they had also been tested in the 
fall of 1974. Therefore, the grade level sample sizes upon 
which post-test means were based, were used as an indicator of 
the number of students in each grade level for whom both pre and 
post-data x^^ere collected. For Manatee elementary schools, grade 
level sample sizes ranged from 32 to 56. For Sarasota elementary 
schools, grade level sample sizes ranged from 27 to 79. Generally, 
the smallest sample sizes appeared at the lov7Gr grade levels. 
Both grades 7 and 8 for Manatee jniddle schools had between 50 
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and 60 students. Sample sizes for Sarasota junior high grades 
ranged from a low of A8 at grade 7 to a high of 58 at grade 9. 
Grades 4 through 9 were sampled at the Pineview School, with 
sample sizes of 2, 4, 3, 6, 9, and 5, respectively. 

With the exception of the grade level sample sizes at Pineview, 
the number of students sampled at each grade level seem reasonably 
adequate for providing rather stable results for the group 
preformance data that were presented for grade levels in the 
USF reports . 

Sample Compos ition 

The relative frequency of various classifications of students 
is presented in this section. Sample composition with respect 
to sex, race, and occupation of head of household will be 
described. 

Sex . Some idea of the relative frequency of males and females 
in the samples was possible by inspecting tables with male and 
female group data shown separately. The proportions of males and 
females in all elementary grades for both counties were quite 
similar. For the Manatee middle schools, there were between 11/2 
to. 2 times more males than females. A slightly larger proportion 
of females than males were included in the Sarasota junior high 
school sample. No information was ctvailable for the male-female 
composition of the Pineview sample. 
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Race . In Manatee County, the samples included approximately three 
timcc? more anglos than blacks. For the elementary schools, this 
proportion appears to hold for all grade levels. On the other 
hand, the Sarasota samples included even larger proportions of 
anglos relative to blacks • Considering all grade levels at the 
elementary schools, the Sarasota sample included approximately 
15 times more anglos than blacks. This proportion, however, varied 
somewhat with grade; grade samples ranged from 8 times more anglos 
to 21 times more anglos than blacks. The Sarasota junior high 
sample included slightly more than 6 1/2 more anglos than blacks. 

As will be evident later in this report, the relative proportion 
of anglos and blacks presents some problems in drawing conclusions 
about group performance in Sarasota. In some cases^ the data on 
black students were based on such- small sample sizes, the data 
should be considered very unstable. 

Occupation . Some data on sample composition were also available 
for one additional independent variable considered in the 
evaluation study. Tables presented in the reports give some ideas 
about the composition of samples with respect to the occupation 
of heads of households. For the Manatee elementary school sample, 
the* highest frequency for occupations of head of household was 
for skilled occupations. In decending order of frequency, this 
was followed by semi-skilled and laborer . occupations. Skilled 
occupations were most highly represented at all grade levels. 
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The highest frequency for the Manatee middle school sample was* 
for the semi-skilled occupations. This was followed, in descending 
order of frequency by skilled and laborer occupations. The data 
on occupation of the head of household for the Middle school 
sample, however, were based on only 65 of the 109 students in the 
post-test study. 

For the Sarasota elementary school sample, the highest frequency 
for parental occupation was for skilled occupations* This was 
followed, in descending order of frequency, by semi-skilled and 
professional occupations. The highest frequency varied by grade 
level, however the highest frequency for a particular grade 
sample was generally one of these three.- For the Sarasota junior 
high students, the highest frequency was for skilled occupations, 
followed by professional and then semi-skilled occupations. 

Sample composition with respect to the occupation of the head of 
household was somewhat different in the two counties. The Sarasota 
county sample included higher relative proportions of students 
whose parental occupations were classified as professional. 

Summary and Comments 

The sample selection process was such that representative samples 
should have resulted. The sample composition data should, there- 
fore, provide reasonable descriptions of the school populations 
in Manatee and Sarasota counties. Although the data also indicate 
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some differences in the student populations in the two counties, 
such differences would not lead one to expect differential student 
outcomes on the basis of population differences alone. 

Sample composition with respect to race and occupation of head 
of household resulted in very small sample sizes for some of the 
claissif ications. With such small sample sizes the data were 
very unstable and it was therefore difficult, in some cases, to 
draw conclusions about performance. 

It is evident that, during the course of the study, students 
were lost from the sample, Pre-test data were not available on 
all the students that were originally selected. Post-data were 
not available for all students who were pre- tested. Even 
though there is no evidence in the USF reports that attritions 
were biased on any of the identified independent variables, the 
rate of attrition and the resultant small sample sizes at post- 
testing produced, in some cases, unstable or misleading post- 
test results. 



Procedures and Instrumentation 



Testing and other data- collection for students in the samples 
in both Manatee and Sarasota counties occurred in October 1974 
and in May 1975. These data collections are identified as the 
pre-testing and post-testing, respectively. The Project Director 
took responsibility for local testing arrangements and test 
administration. The evaluation reports do not specify who 
actually administered the tests and whether there were any 
differences in procedures in the experimental and control schools. 
The preparation of test packets and the orientation of "project 
staff" in the use of the tests, as well as the scoring and 
analysis was done by the Evaluator. 

I ns t rumen t a t i on 

A number of instruments were developed or adopted for use in the 
evaluation studies of Career Education in Manatee and Sarasota 
counties. The following will briefly describe each of the 
instruments used and indicate which instruments were used for 
data collection for each sample. The discussion of the various 
instruments includes references to career education objectives. 
Tnese objectives are the Federal Guideline Objectives reported 
in Appendix A of the Second Quarter Report of A Comprehensive 
Program of Vocational Education for Career Development: K-University . 

Instruments used for both Manatee and Sarasota elementary school 
samples include the following: a) Career Achievement Test, b) 
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Teachers Inferred Self-Concept Scale, and c) Student Information 
Sheet-Elementary, Level I of the Career Achievement Test was 
administered to students in grades K through 2 in both counties. 
Level II was administered to grades 3 and 4. Level III of the 
Career Achievement Test was administered to grade 5 in Manatee 
County and grades 5 and 6 in Sarasota County. 

The instruments administered to grades 6 through 8 in Manatee middle 
schools and grades 7 through 9 in Sarasota junior high schools 
included the following: a) Career Maturity Inventory , b) Rotter's 
Revised Scale of Locus of Control, c) Rosenberg's Self-Esteem 
Scale, and d) Studpnt Information Sheet-Secondary. After pre- 
test data were collected, the Career Maturity Inventory was 
judged inappropriate for grade 6, and therefore the grade 6 
sample in Manatee County was not post- tested and was dropped 
from the study. 

The Career Achievement Test (CAT) was "developed specifically 
for measuring the objectives for career education set forth in 
the Florida Career Education Model and the National Standard 
Career Education Model". The CAT has three levels and each 
level has two forms. Form A of each level V7as used for pre- 
testing and form B for post-testing. Level I was administered 
to grades K through 2, Level II to grades 3 and 4, and Level III 
to grades 5 and 6. Level I contains 30 items. Level II has 40 
items, and there are 38 items in Level III. All items are of 
multiple-choice format. *\ O 
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An ETS review of CAT Level III, Form B indicates the test concerns 
general information about careers and work behavior, and apparently 
it touches lightly on a number of objectives for the Career Education 
Projects in Manatee and Sarasota. The test reviewer found at least 
a partial relationship between the test and the following objectives: 
1. Students will recognize the bases of various work values, 
1. Students V7ill possess positive attitudes towards paid 
and unpaid work, 
' 3. Students will know entry requirements for major types 
of paid and unpaid work, 
4. Students will know the important factors associated with 
various work roles. 

The reviewer, however , concludes that the test (especially if the 
total test score is used) does not seem to relate clearly to 
any specific objective. 

Since no information was found to indicate the equivalence of 
Forms A and B, the results of pre-post data analyses, involving 
one form for the pre-test and the alternate form for the post- 
test, cannot be interpreted unambiguously. A comparable problem 
occurs if one attempts to interpret differences in performance 
,from one test level to another. 

The Teachers Inferred Self-Concept Scale (TISCS) , which was 
developed at a university in Florida, requires the teacher to 
rate a student on 15 items, using a five point scale. The 
TISCS yields a total score (maximum score = 75) in addition 
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to three subscale scores: 

Self = the way the individual sees himself. 

Others = the way others see the individual, 

Community = the way the individual sees others. 
A higher score, in all cases, indicates a more positive perception. 
Teachers of elementary students completed the same form in the 
fall and again in the spring. 

The Career Maturity Inventory (CMI) is a test battery which includes 
an attitude scale (CMI-ATT) and a competence test (CMI-C). The CMI-C 
contains five subtests and yields a score for each subtest. The 
competence total score is the sum of the five subtest scores. The 
subtests are as follows: CI) Knowing Yourself, C2) Knowing 
About Jobs, C3) Choosing a Job, C4) Looking Ahead, and C5) What 
Should They Do? These subtests may be thought of as providing 
measures of self-appraisal, occupational information, goal 
selection, planning, and problem solving. In each of the 
competence subtests, hypothetical situations are presented and 
the student is asked to choose one of five alternatives. 

The '^attitude scale ''elicits the feelings, the subjective reactions, 
the dispositions that the individual has toward making a career 
choice and entering the world of work". The ETS review found a 
relation between the attitude test and the following career 
education objective: "Students will demonstrate active involvement 
in career decision-making". The CI subtest was found related 
to "Students will be able to describe their own current abilities 
and limitations". The C2 subtest relates to "Students will know 
the major duties and required abilities of different types of 



-16- 



paid and unpaid work". The C3 subtest relates to the following 
objective: "Students will be able to associate their own 
abilities and limitations with possible success in present or 
future paid and unpaid work". The subtest C4 relates to "Students 
,will know the steps to be taken and the factors to be considered 
in career planning". The C5 subtest was found to be related to 
both of the following objectives: "Students will be able to 
demonstrate generally useful decisidn-making skills", and 
"Students will be able to identify, locate, and utilize sources 
of information to solve career decision-making problems". The 
reviewer, however, concluded that the subtests don't measure 
these objectives exactly, and at least for the attitude scale, 
the total score reflects a combination of "different, and possibly 
unrelated objectives". Only one form of the CMI is available, 
therefore students were administered the same form in both fall 
and spring. 

Rotter's Revised Scale of Locus of Control is a measure of the 
degree to which an individual views what happens to him as being 
related to his own actions (internal control) versus being 
related to other forces such as luck, chance, powerful others, etc. 
(external control). The scale consists of eleven items, each of 
which is given a score of one to four. Lower scores indicate 
internal control and higher scores indicate more external 
control. The locus of control measure relates most strongly to 
"Students will have a positive attitude towards responsibility for 
their ov/n behavior and accomplishment of self-imposed work tasks", 
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and it is suggested that it also relates, somewhat peripherally, 
to different aspects of self-awareness . In some of the USF 
reports this measure is also referred to as Rotter's Scale of 
Internal-External Control. The USF reports include group per- 
formance data for a variable called locus of control, operation- 
ally defined as the total score on the Rotter's scale. Additional 
information regarding this instrument is available in the ETS 
test review. 

Rosenberg's Self-Esteem Scale is a ten item self-report instrument 
which indicates the student's attitude toward himself , and as 
such, relates to the objective ^'Students will display positive 
attitudes toward themselves". Each item is answered on a four 
point scale ranging from strongly agree to strongly disagree. 
Although each item has four response options, the items are 
scored either positively or negatively. Some of the items are 
combined such that total scores range from 0 to 6. Higher scores 
denote a higher self-esteem. In some of the USF reports this 
measure is also referred to as Rosenberg's Self-Concept Scale. 
The USF reports also include group performance data, for middle 
school and junior high school samples, on a variable ca:lled 
self-concept, operationally defined as the score on the Rosenberg's 
Scale. 

The Student Information Sheet - Elementary (SIS-E) is an eleven 
item questionnaire used to obtain background and other information 
about students in the sample* Completed by the teacher, the 
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questionnaire asks about a student's grade, sex, and race. Other 
items ask for the occupation of the head of the household, the . 
student's IQ score, and the student's reading score. In' addition, 
the teacher was asked to rate the student, on a ten point scale, 
on three variables: a) self-concept, b) academic performance, 
and c) interest in school. It should be pointed out that, although 
IQ and reading scores were requested, these data were not included 
in the analyses. It is possible that the evaluator questioned the 
validity of the data, as would this writer. However, it is also 
possible that teachers were not willing and/or able to provide 
the requested information. 

The Student Information Sheet - Secondary (SIS-S) is a two part 
questionnaire used to obtain background and other information 
about students in the middle school and junior high samples. The 
first part consists of nine items completed by the student and 
items ask for information about the student's sex, race, length 
of residence in the county, plans to attend college, attitude 
toward school, work experience, and certainty about the future. 
The second part, completed by the student's teacher, asks for 
the occupation of the head of household, the student's IQ score, 
and the student's reading score. Previous comments regarding the 
IQ and reading scores requested on SIS-E apply here also. The 
reader is referred to the ETS test reviews for additional comments 
on the SIS-S. 



Comments and Recommendations 

The TISCS was administered in both the fall and spring. Although 



group performance on the TISCS is presented tabularly in the USF 
reports, no analyses of covariance were performed using data 
from this instrument. One cannot help but wonder why these data, 
as well as other teacher report data for elementary students 
(see SIS-E) , were not included in the covariance analyses. 

Both the SIS-E and SIS-S requested information which was not 
included in covariance analyses and which did not appear in any 
group performance tables in the USF reports. If data are not 
to be used, or if questionable validity and reliability is indicated, 
there are no apparent reasons why requests for such data should 
be made. It is also suggested that one should look very carefully 
at the reliability of items such as those included on the 
Student Information Sheet. 

The matching of tests and test items to career education objectives 
is an important problem requiring attention. This is an ever 
present problem whenever there is an attempt to use "off the 
shelf" tests.. For both the CAT and the CMI, the ETS test reviews 
suggest scoring only those items that relate directly to an 
objective and/or tailoring specific subscales for the various 
objectives measured by the test. If no other reasonable option 
presents itself, consideration might also be given to reporting 
performance separately for each item identified as a r-^tch to 
a particular objective. Probably only as a last resort should 
new tests be developed; such a process is generally too expensive 
and time consuming to be feasible for reasonably small projects. 

The ETS reviews provide suggested alternatives to the instruments 
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used in 1974-75. Instruments used in the 1974-75 evaluation, as 
xvrell as other available instruments measuring similar concepts, 
have also been reviewed by ETS in preparation for the FY 76 
evaluation of the project. That review process included not 
only the examining the evidence of technical quality for the 
instrument, but also examining the match between career education 
objectives and items or item sets, and subtests or scales. Such 
an examination has obvious implications, insofar as allowing 
interpretation of results as indicating the effect of career 
education program efforts, or indicating whether the observed 
results match the expected results if career education objectives 
were met. It is possible that the problems of the match between 
instruments and objectives explain why the USF reports included 
discussion of results only in terms of performance on instruments, 
and did not include discussion of results in terms of specific 
career education objectives. It is reasonable to expect that 
all objectives will not be met to the same degree, and that it 
would be important to know where those differences occurred and 
whether that differences match differences in the "amount of 
effort" directed toward those objectives. 



-21- 



Variables and Analyses 

The major concern identified in the USF reports x^as one of deter- 
mining whether students in the experimental schools demonstrated 
higher performance on the post-test than students in the control 
schools. Experimental and control schools were identified by 
the Career Education Project Director and members of the adminis- 
trative staff. Control schools were selected on the basis of 
being similar to the experimental schools in terms of "the 
ability of the students and the quality of the staff". The 
Pineview school study involved pre-post test comparisons without 
a control group. 

In addition to attempts to investigate experimental versus control 
group differences, the USF reports indicate that attempts were 
made to determine if sex, race, or occupation of head-of-household 
had an effect on performance on the post-test. For the Sarasota 
samples, the data were also analyzed to determine if involvement 
of the students in the project (i.e. "involvement") had an effect 
on performance on the post'-test. This writer has not been able 
to determine the source of data for the variable involvement of 
the students in the project, however, the USF reports indicate 
that the classification on this variable was different from the 
experimental vs. control group classification. These variables 
were considered the independent variables in a series of analyses 
of covariance. It is not obvious why these specific variables 
were selected as the independent variables, but it will be assumed 
that there are theoretical., empirical or programmatic reasons 
for their selection. '^^ 
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Tlie source of data on sex, race and occupation of household was 
the Student Information Sheet. Apparently^ so few students fell 
into racial classifications other than anglo or black, that only 
those two classifications were used as levels of the variable 
race, when analyses of covariance were performed. 

Because differing levels of the Career Achievement Test were ad^ 
ministered to elementary students, the elementary school sample 
was divided into grade groupings for purposes of analysis (i.e., 
analysis samples). This resulted in three analysis samples for 
each elementary sample. For Manatee elementary schools, the 
analysis samples were defined as a) K through grade 2, b) grades 
3 and 4, and c) grade 5. For the Sarasota elementary school 
sample, the analysis samples were a) K through grade 2, b) grades 
3 and 4, and c) grades 5 and 6. 

The dependent variable (s) for each analysis sample varied as 
a function of the grade level (s) of the students in the sample. 
In all cases, the fall score on the variable was used as the 
covariate and the spring score was considered the dependent 
variable. 



Eleven dependent variables were included for analyses involving 
middle school and junior high school samples. Student data on 
the like for school and the certainty about the future variable 
were ol^teined from the Student Information Sheet-Secondary. 
Students indicated their like for school by choosing one of the 
following options: 1) a lot, 2) it^s OK, 3) mildly dislike, or 
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4) strongly dislike. It is assumed that the data were analyzed 
using a system which assigned a 1 to the response "a lot", 2 to 
*'it*s OK*, etc. Certainty about the future was assessed by 
analyzing student's responses to the question "How .sure are you 
about what you want to do in the future in terms of • work, marriage, 
college, etc?" Very sure was assigned a 1, sure a 2, 3 referred 
to not too sure, and 4 indicated very undecided. Lower scores 
for the like for school variable indicated more like for school, 
and lower scores for the certainty about the future variable 
indicated greater certainty. 

From the administration of the Career Maturity Inventory, seven 
scores were obtained and analyzed. The total score on the CMI-ATT 
was used as one dependent variable. The total score for the CMI~C 
plus five subscores were separately analyzed as dependent variables. 
The subscores refer to the five competence subtests previously 
described .in the section on Procedures and Instrumentation. The 
locus of control dependent variable was operationally defined in 
terms of the Rotter's Revised Scale of Locus of Control. Lower 
total scores on this scale indicate more internal control. The 
self -concept dependent variable scores were obtained from the 
Rosenberg's Self-Esteein Scale. Higher scores denote a higher 
self-esteem. 

For each analysis sample, each independent variable was separately 
combined V7ith each dependent variable, and analyzed using analysis 
of covariance. For Manatee elementary school samples, this 
resulted in 12 separate analyses. Using four independent 



variables and eleven dependent variables, 44 analyses were 
applied to Manatee middle school sample data. Fifteen and 55 
analyses V7ere completed for dauj " » Sarasota elementary school 
samples, and the Sarasota junior high sample, respectively. A 
total of 129 analyses of covariance were completed for Manatee 
and Sarasota County student samples • 

For Piaeview students, pre-post comparisons were completed for 
each of the following measures : CMI-ATT, CMI-Cl, CMI-C2, CMI-C3, 
CMI-C4, CMI-C5, CMI-C Total, Locus of Control, and Self-Concept. 
It is not clear whether statistical analysis techniques were 
applied in making pre-post comparisons, nor is it clear whether 
data from all or some grade levels were combined for purposes 
of making the comparisons. 

With the exception of the variable "involvement", it seems 
reasonable to suspect an interaction effect for the independent 
variables. Of special importance would be any interaction of 
each independent variable and the group membership (i.e., 
experimental vs. control). Sample sizes were probably not large 
enough to allow the use of a factorial design in which all indepen- 
dent variables were used in the same analysis. It might have 
been possible and, in this V7r iter's view, preferable to do two- 
V7ay analyses of covariance in which group was always one of the 
independent variables. If interactions were present, the one-way 
analyses, as used in the US? study, v7ould not detect them. 

Unfortunately, this V7riter was unable to determine the meaning 
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of the variable "involvement of the students in the project", A 
first thought is that it defines degree of exposure to career 
education programming efforts. If this is the case, the question 
arises as to whether only experimental group students were 
classified according to this variable or if both experimental 
and control group students were so classified. Because of "spill 
over effects" and "diffusion effects". of implementing new programs, 
such a variable might be more meaningful than i Vc.riable having 
only t,^o levels, i. e., one level referring to "experimental" 
and one referring to "control". 

Group performance data were calculated and presented tabularly 
in each report. These data were organized in several different 
vrays "to facilitate the detection of differing levels of • * • • 
performance for differing types of students". Group performance 
data were calculated and presented for different types of students, 
by using the following student description variables: a) grade 
level, b) sex, c) race, d) occupation of head of household, e) 
college plans, f) like for school, g) employment experience, and 
h) certainty about the future. No data were presented for. 
experimental vs. control group performance. A total of 146 
tables were presented for the pre-test and post-test performance 
of students in the Manatee and Sarasota samples. The samples 
for v7hom data are presented in the post-test results reports 
ai?e not identical to those for whom comparable data are presented 
in the pre- test results report. In some cases, this situation 
made it impossible to draw conclusions about the pre to post- 
test stability of relationships found in the fall of 1974. 
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Results and Conclusions 

The results section of each report on pre-test results included 
the presentation of group performance data tables and ,a brief 
description of "notable differences" and interesting relationships 
indicated from inspection of the data. Summary conclusions 
appeared in each report. Some of the conclusions ' presented in • 
the USF reports are also discussed in this review, and an attempt 
has been made to integrate the conclusions from the separate 
counties. Attention is also given to USF conclusions for which 
noncritical acceptance might not be appropriate. 

Group data for the post-test performance were also calculated 
and presented. The tabular organizations was practically the 
same as that used for presenting pre-test data. In the post- 
test reports, however, these data were not described and 
conclusions were not reached and presented. This report attempts 
to examine, based on both the pre-test and post-test data, whether 
relationships and patterns of performance observed in October 
remained constant during the 1974-75 year. 

USF Results and Conclusions 

The reported results and conclusions in the USF post-test 
reports were based on the analyses of covariance that were 
performed. Although no tables were presented to show the results 
of these analyses, the report does indicate how many and which 
•ones of the analyses showed statistically significant effects. • 
The USF results for Manatee and Sarasota samples are indicated 
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below, including the results reported for the pre-test post- 
test comparisons completed for the Pineview students. 

Manatee Elementary Samples , Of the 12 covariance analyses 
conducted, tv/o showed a statistically significant effect at p < .01. 
In both cases, race was the significant source of variation. 
Anglo students performed significantly higher than black students 
on the Career Achievement Test. This difference was found when 
data from the kindergarten - grade 2 sample were analyzed and 
also when data from the grade 3 and 4 sample were analyzed. 

Sarasota Elementary Samples , Fifteen analyses of covariance 
were conducted and one shov/ed a statistically significant 
difference at p < .01. The kindergarten - grade 2 experimental 
group sample performed significantly higher than the kindergarten- 
grade 2 control group sample. This v/as the only case, reported 
in which experimental versus control group differences x^ere 
found. 

Manatee Middle School Sample. T\<jo of the 44 covariance analyses 
indicated a statistically significant difference in performance 
for the groups being compared (p < .01). Anglos performed 
significantly higher than blacks on the Career Maturity Inventory 
Attitude Scale, and females performed significantly higher than 
males on OEL-Cl. 

Sarasota Junior Hi^h Sample / None of the 55 . cinalyses of 
covariance showed a statistically significant effect. 
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Sarasota Pineview Sample (s) ♦ Actual pre-test versus post- 
test performance data were not presented in the USF final report; 
only post- test group performance data were presented. The report 
does indicate there were '*no clear, consistent differences between 
performance on the pre-tests and that on the post-tests. The 
same general patterns of response observed on the pre-test were 
observed on the post-test." 

The USF final reports for Manatee and Sarasota counties contain 
identical conclusions: "there were no apparent differences in 
performance of students exposed to career education and those 
not exposed to career eeucation". The report continues, "The 
most immediate explanation of these results is that being exposed 
to career education does not effect students in significant ways. 
However, there are some alternative explanations which should be 
considered". These alternative explanations identified in the 
USF reports are presented below: 

1, The control schools might have also exposed their 
students to career education, so that control- 
experimental group comparisons would not reflect 
the effects of exposure to career education. 
2* Career education may have had an effect on students, 
but not in terms of the objectives measured by the 
instruments used in the evaluation, 
3, Career education may never have been instituted in the 
experimental schools, such that students never really 
received the "treatment". 

Go 
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4, The students may have received the "treatment", but 

the time period may have been too short to produce any 
significant change. Differences may appear after* two 
or three years which were not apparent this year. 

With 129 analyses and p < ,01 set for the significance, if all 
analyses can be considered as belonging to the same experiment, 
at least one significant effect could occur by chance. Although 
the USF analyses resulted in one significant effect for group, 
the evaluator rightly concludes that there were no apparent differ- 
ences in ^performance of students in experimental and control 
groups. 

It seems appropriate, at this point, to coiment on the alternative 
explanations for the results that were identified in the USF reports. 
Explanations //l and //3 are closely tied together and refer to what 
this writer would call the "existence and integrity of the indepen- 
dent variable", where the independent variable is' group. Without 
variance a variable does not exist, and without systematic 
variance relationships cannot be observed. In order to examine 
the effects of a treatment, it is necessary to establish, by . 
some appropriate means, that the treatment is, in fact, occurring, 
l^ether '^spill over" has occurred or not, if one wants to 
evaluate the possible differential effects of being exposed to 
a treatment as opposed to not being exposed to that treatment, 
it is necessary to establish that the two experiences are, in 
fact, different in some meaningful way(s). 
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It is quite possible that the ''degree of implementation" of the 
treatment may vary within the treatment group (e.g., teacher 
variation, school variation, etc.) and/or that exposure to varying 
degrees of the treatment has occurred in both "experimental" 
and "control" groups. Perhaps, instead of defining experimental 
and control groups, a better procedure might involve defining a 
number of treatment groups, based on variation in the degree or 
type of treatment exposure. Ifliatever procedure is used, however, 
it is necessary to establish that a treatment is occurring, and, 
if treatment group is a variable, that the treatment differs from 
one group to another. At the very least, a description of the 
"treatment" in each group would be needed, in order to interpret 
"resulting" differences or lack of differences in student 
performance. 

All four alternative, explanations listed in the USF reports, 
represent plausible hypotheses for explaining the rp,sults. Future 
studies should certainly involve attempts to eliminate or reduce 
the plausibility of these explanations as rival hypotheses. 

Ad ditional Interpretations and Coiiclusions 

Additional analyses, interpretations and conclusions with respect 
to the "treatment" group variable are not possible because 
separate experimental and control group data were not available 
in the USF reports. It is possible, however, to provide additional 
information on other variables that should be helpful in inter- 
preting and extending that provided by the USF reports. 

ERIC 
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Because of differences in the population and the career education 
programs in Manatee and Sarasota counties, no attempt was made 
to combine results directly. However, when similar relationships 
and patterns of performance are observed for both comities, it 
seems reasonable to draw a single conclusion that could be 
expected to apply to either or both counties. 

The USF pre~test results reports suggest that pre-test results 
would be used for program planning decisions. If such decisions 
were made, two questions could be asked. First, do the data 
support the initial conclusions upon vjhich decisions were made. 
Secondly, were the resulting programming activities "successful" 
in changing the patterns of performance observed in the fall. 
The following presentation will therefore focus on the validity 
and stability of relationships between variables and patterns 
of performance that were reported in the fall of 1974. 

Results and Conclusions for Elementary School Samples 

Patterns of performance for the elementary school samples are 
discussed below. Validity and stability findings are organized 
around main headings that were used for presentation of results 
in the USF reports. 

Patterns of Performance by Grade Level . For both Manatee and 
Sarasota elementary samples, both pre and post-test data indicate 
no grade level differences for teacher ratings of students on 
self-concept, interest in school, and academic achievement, nor 
for teacher ratings on the TISCS. For both counties, however, 
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the USF pre-test results reports state that performance on the 
Career Achievement Tests increases with higher grade levels. 
An inspection of Table 1 in each pre-test report reveals that 
the mean value for CAT performance for a grade level sample 
was not always larger as the grade level increased. One 
should remember, however, that K - 2nd grade students took 
Level I, 3rd and 4th graders took Level II, and 5th and 6th 
grade students took Level III of the CAT. Within Levels I, II, 
and III, the mean CAT score increased as grade level increased, 
for both counties. This increase, although very small in some 
cases, was found for both the pre-testing and post-testing 
scores. 

Patterns of Performance by Sex * On all measures, the relation- 
ship between sex and performance remained ronstant during the 
year. In the fall and again in the spring, teachers in both 
counties rated girls higher than boys on every measure on the 
SIS-E and the TISCS. The lack of significant effects for sex 
in the analyses of covariance also indicates that the relation- 
ship for the CAT did not change from the pre-test to the post- 
test. Third and fourth grade Manatee females performed higher 
than males on the CAT; no sex differences occurred at other 
grade levels. Sarasota males, however, scored slight' ly higher 
than girls on the CAT at five grade levels. 

Patterns of Performance by Race . In the fall of 1974 and again 
in the spring of 1975 ^ teachers in both counties rated anglos 
higher than blacks on every measure on the SIS-E and the TiSCS. 

■ . ' G7 



-33- 



The USF reports indicate that on the pre- test Sarasota and Manatee 
anglos performed higher than blacks at every grade level on the 
CAT. In addition, in Manatee, the differences in scores were 
found to become larger at the higher grade levels. The analyses 
of covariance indicated a significant effect, in Manatee county, 
for race on the CAT Level I (grades K - 2) and Level II (grades 
3 and 4). For grades K - 4, in Manatee, the initial pre-test 
difference between anglo and black students was even greater 
on the post-test» Although race was not a significant effect 
for the other Manatee and Sarasota samples, the post-test anglo- 
black mean differences were also larger than the pre-test anglo- 
black mean differences. 

Patterns of Performance by Occupation of the Head_^of Household . 
The USF studies reported that, in both Manatee and Sarasota 
counties, students from ^'higher" occupational groups were rated 
more favorably on the SIS-E and TISCS measures, and they scored 
higher on the CAT. The group performance data generally support 
that statement of the relationship between occupation of the 
head of household and performance on the CAT pre-test. There 
are exceptions, however, but most are associated with quite 
small sample sizes (and therefore relatively unstable measure- 
ments). The lack of significant effects for occupation in 
the analyses of covariance indicates the relationship between 
occupation and CAT performance did not change as a result of 
experiences intervening betv7een pre and post- testing . A similar 
lack of change occurred for ratings on the SIS-E and TISCS measures. 

Bo 
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Results and Conclusions for Middle and Junior High Samples 

Patterns of performance for the Manatee middle school students 
and the Sarasota junior high students are discussed below. 
Validity .a^id stability findings are organized around headings 
that were used for presentation of results in the USF reports. 

Patterns of Performance by Grade Level . For both the pre- 
test and the post-test, performance on the various sections 
of the CM! generally increased with increases in grade level. 
Grade level performances, however, were quite similar for 
the locus of control and self -concept measures. These relation- 
ships betv/een grade level and performance on the various measures 
were observed for both the Manatee middle school and the Sarasota 
junior high samples. 

Approximately 80% of all students at each grade level in both 
counties planned to attend college in the fall. With the exception 
of Manatee 7th graders, the USF reports indicated small propor- 
tions of students in each grade level planned to attend college 
in the spring. Because of changes in the sample from fall to 
spring (especially loss of students) this may not reflect any 
difference in student's plans, but only differences in the saTTiple* 

Although the USF studies reported no notable grade differences 
in like for school for Sarasota students, there is a statement 
that for Manatee County ^ 7th graders had the least like for 
school. Data included in Table 3 of the pre-test report suggest 
:**.\t the least like for school occurred for 6th graders rather 
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than 7th graders. In both counties, there was some tendency for 
the students, whose data are included in the post-test report, 
to express less like for school. It should be emphasized 
again, however, that the spring sample size is smaller and the 
fall to spring differences may more closely reflect differences 
in the sample than differences or changes in students ' attitudes . 

Patterns of Performance by Sex * Manatee and Sarasota females 
scored higher than males on all sections of the Career Maturity 
Inventory. The relationships between sex and performance on 
the CHI remained relatively constant for all cases except the 
one involving Manatee students and the CMI-Cl. On CMI-Cl, 
females performed significantly higher than males on the post- 
test even after the pre-test scores were adjusted for initial 
differences. 

The USF reports state that Manatee males scored higher than 
females on the locus of control and self-concept measures, and 
Sarasota males scored higher on the. self-concept measure. Based 
on additional analyses, however, it was found that Manatee 
females scored significantly higher (p <.05) than Manatee males 
on the locus of control, and the male-female difference on the 
self-concept measure was not significant. The Sarasota conclusions 
are supported by the data. Because the analyses of covariance 
resulted in no significant effects for sex on these measures, 
it is then possible to conclude that the relationships just 
described for' the pre-test data remained constant from fall 1974 
to spring 1975. \ 
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The relationship between sex and expressed like for school and 
bet^i7een sex and exprevSsed certainty about the future, can be 
considered constant during the 1974-75 school year. The lack 
of significance in the analyses of covariance support such a 
statement. The pre-~test data indicate Manatee females expressed 
more like for school than Manatee males, but Sarasota males and 
females expressed approximately the same degree of like for 
school. In both counties, males expressed more certainty about 
their future than females . 

Patterns of Performance by Race » The only situation in which 
the relationship betx^reen race and the dependent variable did 
not remain constant during 1974-75 was for Manatee middle 
school students' performance on the CMI Attitude scale. Al- 
though, on the pre-test and post-test, anglos scored higher 
than blacks, in both counties, this initial difference still 
did not account for the difference on the post CMI attitude for 
Manatee. There were no apparent differences between anglos 
and blacks on the locus of control and self -concep t pre-tests. 
This relationship was maintained and validated on the post- 
tests. 

In the fall of 1974, Sarasota black students expressed more 
like for school than their anglo classmates. Racial differences 
in like for school in Manatee were not quite so striking and 
possibly indicate no 'real difference. The relationship betv7een 
race and expressed like for school was apparently not changed 
as the result of experiences during the school year. It should 
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be noted, however, that the information on like for school for 
Sarasota blacks V7as based on only 21 students whereas there were 
157 angles in the sample. 

Sarasota black students expressed more certainty' about their 
future than anglo students. In Manatee there were no striking 
differences in certainty about the future. These relationships 
which were observed on the pre- test remained constant for the 
post-test. ' ' 

Patterns of Performance by Occupation of the Head of Household . 
Sarasota students whose heads of household were in the "professional" 
or "skilled" groups performed higher on the CMI pretests than 
students from other occupational groups. Manatee students whose 
heads of household were in the "skilled" group demonstrated the 
highest performance on the CMI pretests. These relationships 
between occupation of head of household and CMI tests remained 
constant from pre- test to post- test. 

A large percentage of the students in each occupational category, 
in both counties, expressed like for school. The results of the 
analysis of covariance indicate that experiences during 1974-75 
did not affect the relationship between occupation and like for 
school that existed in the fall of 1974. 

Summary . In general the data support the USF pre- test conclusions. 
Only tv^o situations v;cre identified for V7hich the data were clearly 
not in agreement with those conclusions. Both situations involved 
conclusions about Manatee middle school students. In one case, 
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the data were not in agreement V7ith the USF conclusion regarding 
the grade level with the least like for school. The other case • 
involved a conclusion about the relationship between sex and scores 
on the locus of control and self --concept measures. 

* For a number of the relationships examined and discussed, for both 
pre^test and post-test data, group differences were quite small. 
Even though small differences may be statistically significant, 
the question of whether those small differences are meaningful 
differences cannot be answered by statistical analysis. 

With minor exceptions, the relationships between variables and 
the patterns of scores, that v/ere present at the time of pre- 
testing were also found to be present at the time of post- 
testing. This means that the experiences provided for students 
during 1974-75 did not change their patterns of performance. In 
each case V7here a significant effect was found in the analysis of 
covariance, the group that scored higher in the fall also scored 
higher in the spring, however, the spring differences in per- 
formance were substantially greater than the fall differences. 

The descriptions, interpretations, and conclusions presented 
in this ETS r'feport serve to supplement and extend those provided 
by the University of South Florida reports. The ETS conclusions 
are quite consistent with those reached by the USF evaluator. 
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Career Education Achievement Test (level III, Form B) 



General description ; 



Adminis tration time ; 

Scoring time: 

Grade levels for 
\</hich instrument 
is appropriate; 

Technical quality: 



Relationship to 
program objectives: 



Use of subscales: 
liecomnendation : 



Suggested alternatives: 



This 38 item multiple choice examination concerns general 
information about careers and work behaviors. Although 
there are other levels and forms of this test, only level 
III, Form B will be reviewed. 

Approximately 20 to 30 minutes. 

About 2 to 3 minutes per answer sheet if scored. by hand. 

Language seems to be geared at about an eighth grade level 
although younger grades could understand the items and 
responses if the exam were administered orally. 

This instrument is unpublished, therefore, no information 
regarding its reliability or validity is available. 

The test does not seem to relate clearly to any specific 
objective but appears to touch lightly a number of objectives • 
The test relates to both objectives A and B under III 
although the total score for the test probably does not 
adequately measure accomplishment of these objectives. There 
is also a partial relationship between the test and 
objectives IV-C and E. 

None are present. 

One of the main problems with this test is interpreting 
its results. If student A scores higher than student B, 
that does not necessarily mean that student A possesses 
more "positive attitudes towards paid and unpaid work," 
nor does it mean thatstudent A will have a more "positive 
attitude toward the concep.t of quality in relation to a 
work task." The problem, of course, is that the test is 
a general paper and pencil measure and the program 
objectives it attempts to measure seem to require specific 
observational or performance measures. For example, 
objectives like IV-C and E can only be measured by direct 
observation. On the other hand it is extremely difficult 
to measure objectives like A and B under III in any way 
other than by individualized assessment. Many of the items 
are trivial - e.g., numbers 6, 7, 17, 23, and 29. Other 
items appear somewhat controversial - e.g., 5 and 29. 
This test should be substantially revised, eliminating the 
trivial items, constructing better item responses - in 
some cases you can tell the correct answer without reading 
the item - and tailoring specific subscales to the program 
objectives . 

Suggested alternatives are listed under Career Maturity 
Inventory . 
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General description: 



Adminis t ration time : 



Scoring time: 

Grade levels for 
which instrument 
is appropriate: 

Technical quality: 



Relationship to 
program objectives: 

Use of subscales : 

Recommendation : 



Teachers inferred self-concept scale 



This is a 30 item scale where teachers note the incidence 
of student behavior on a five point scale ranging from 
'^never" to **always.'* The behaviors described are said to 
relate to the students self-concept. Typical items on which 
students are rated include: "Thinks. he is right," "Appears 
unsociable, " and "Is defiant . " 

The teacher completes one form for each student. It would 
require a teacher 5 to 10 minutes to conscientiously answer 
the questionnaire for a student. 

About 1 minute per form. 

The scale was originally developed for students in grades 
one through six. 



\7hen counselor and teacher ratings (total scores) for 180 
students were correlated, the coefficient 'found was .50. 
Split-half reliabilities of .86, .86, and .90 have been 
found by counselors, teachers, and both combined. Co- 
efficient alpha was .92 for counselors and ,91 for teachers. 
On the other hand, test-retest reliabilities with a six- 
month interval has been found to be .66. Judges rated 100 
items on . their appropriateness as self-esteem indicator. 
Six of the eight judges agreed on 37 items being appropriate. 
Seven items were eliminated as being redundant. Scores 
have been found unrelated to age, but V7ere weakly related to 
scholastic achievement, intelligence, and negatively related 
to competency. 

This self-concept measure relates to objective I-C. 



No subscales are available. 

Attempting to measure self-esteem through observed behavior 
is a noble enterprise in* general, however, the behaviors 
that are rated in this scale are not sufficient indicators 
of self-esteem. The scale may be measuring something 
entirely different from self-esteem. Also, teachers are 
not good raters of these behaviors. These types of ratings 
get at the exact things that the Buckley amendment is 
designed to counteract. Rosenberg's scale, which can be 
administered and scored by the student himself, would be a 
better choice. 



Suggested alternatives: See suggestions regarding Rosenberg's scale. 



-3- 



Caroer Maturity Inventory 



General description: 



This test battery includes an attitude scale and competence 
test which contains five subtests: self-appraisal, 
occupational information, goal selection, planning, and 
problem solving. The attitude scale "elicits the feelings, 
the subjective reactions, the dispositions that the individual 
has toward making a career choice and entering the world 
of work." In all, five factors of career choice attitude 
are measured: involvement in the choice process, orienta- 
tion toward work, independence in decision making, preference 
for career choice factors, and conceptions of the career 
choice process. 



Adminis tration time : 
Scoring time: 



In each of the competence test subtests, a number of 
hypothetical situations are presented in each instance and 
the respondent is asked to choose one of five alternatives. 

2-1/2 hours are required for the complete inventory. 

Inventories can either be scored by machine or by hand 
with a scoring stencil or answer key. It is estimated that 
one or two minutes are required to score one inventory. 



Technical quality: 



Relationship to 
program objectives; 



EKLC 



The inventory was constructed in a very systematic manner. 
The internal consistency estimates for the attitude 
scale have been calculated for grades 6 through 12 and 
average .yA. One-year test-retest reliability was found to 
be .71. Internal consistency estimates for the competence 
tests have been in the .70s and .80s. Tlie attitude inventory 
was constructed from a pool of items that were theoretically 
relevant, and "linguistically representative of the verbal 
vocational behavior of adolescents." Items were selected from 
the pool that differentiated among age and grade levels. 
Content validity was achieved by having judges indicate what 
they considered as the most mature response. Judges had 74 
percent agret ^nt with the standardization sample. The 
construct validity of the competence test has been explored 
by obtaining correlations among subtests. The r's ranged 
from .25 to .73, V7ith a mean of .54. 

The attitude scale relates to objective V-F; the subtest 
"knowing yourself" relates to objective I-A; the subtest 
"knowing about jobs" relates to objective IV-A; the sub- 
test "choosing a job" relates to objective V-A; the subtest 
"looking ahead" relates to objective V-E; and the subtest 
"what should they- do?" relates to both objectives II-D and 
V-C. Of course the problem is that the inventory scales do 
not measure the objective exactly as stated. For example, 
the attitude scale has several items dealing with whether 
a student is actively involved in career decision making. 
But it also has items in the scale that are unrelated to 
involvement with career decision making. The total 
score for the attitude scale does not reflect the desired 
objective, but rather a combination of different, and 
possibly unrelated, objectives. / , 
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Use of subscales: The Career Maturity Inventory gives a score for each of 

the subtests. The six scores are placed in a profile. 

Hecoininendatioas : Tlvis type of assessment instrument does not, and indeed 

cannot, measure accomplishment of the program objectives. 
The data presented in the Theory and Research Handbook 
regarding the validity of the inventory scales is lacking. 
There are also a nuni^er of other problems, such as using 
empirically keyed items in the attitude scale and the 
apparent lack of correspondence between the items and 
the scale names. 

Many assumptions were made regarding "career maturity** 
that are untested, Tne competence test, for exanple, 
assumes that ^^individuals v7ho can accurately appraise the 
career relevant capabilities of others are good self- 
appraisers," .The attitude scale apparently is multi- 
dimensional, yet scores are reported on a one-dimensional 
scale, which is misleading. 

On the other hand, it is extremely difficult to 
construct a test V7ith the aims of the Career Maturity 
Inventory. The author, John Crites, is a respected 
individual who has devoted much of his work to vocational 
and career development, and as such, lends a great deal 
of authority to the inventory. However, it is suggested 
that to assess career education objectives administer the 
whole inventory to students, but score only the items that 
relate directly to the objective being measured. Of course, 
this v/ould destroy the norms and lower the reliability, 
but the assessment would be improved. 

Suggested alternatives: There are no good ways of solving 'the problem of matching 

test items to career education objectives. Consider 
Super and Forest's Career Development Inventory , Prediger, 
VJestbrook, and Roth's Assessment of Career Development , 
and an old, somewhat out-of-date test by Katz, the ETS 
Guidance Inquiry . Trying to develop local tests is 
discouraged; however, parts of existing tests can be used for 
the assessment. 



Rotter's revised scale of focus of control 



General description: This scale consists of eleven items from an earlier scale 

developed by Rotter.'- TTie scale measures the degree to 
which a student views what happens to him as being related 
to his own actions (internal control) versus being related 



•'^Rotter, J. Generalized expectancies for internal versus external control of 
reinforcement. Psycho logical Monographs , ]966, ^0. 
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to other forces such as luck, chance> powerful others, etc, 
(external control). High scores on this scale denote 
higher external control. The iteiiis selected from Hotter' s 
scale v/ere selected on the basis of being more general, 
adult-oriented, and vork related. • Each item is given a 
score of one to four. 



Adminis tration time : 



Approximately 10 minutes and can be administered on 
individual basis. 



Scoring time: 

Grade levels for 
which instrument 
is appropriate: 



Approximately 1 minute. 



High school juniors and seniors 



Technical quality: 



Although there is quite a bit of information available 
on Rotter's 29 item scale - reliability is about .70 - 
there is no information regarding the reliability 
and validity of this 11 item scale. Supposedly^ 
there have been findings that low scores (internal 
control) are associated with higher status occupations, 
knowledge of the work world, and progress on the job. 
No norms are available. 



Relationship to 
program objectives: 



The focus of control measure relates most strongly to 
objective J.V-E, "positive attitude tov/ard responsibility 
for (the students) o\m behavior" although it also relates, 
though somewhat peripherally, to different aspects of 
self -awareness . 



Use of subscales: 
Recomendation : 



Sugges ted alternatives : 



There are no subscales. 

There is so little information available on this short 
form that it is difficult to make any recommendation. 
Nonetheless, Rotter's scale has been used in many 
important studies even though there have been problems 
of social desirability and multidimensionality in the 
items. Factor analyses of the 29 item scale have 
uncovered one factor, named "personal control," comprised 
mainly of items phrased in the first person. A second 
factor, "control ideology," has been found comprised of 
items phrased in the third person. 

Consider the full 29 item scale rather than the shorter 
form. Another alternative that might be worth noting would 
be to take the full 29 item scale and use only the items 
phrased in the first person - i.e., the "personal control" 
items as a measure for objective IV-E. 
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Rosenberg's self-esteem scale 



General clescr-it)tion: 



Adnvinist ration time : 
Scoring time: 



This scale consists of ten iteins each ansv/ered on a four 
point scale ranging frorn strongly agree to strongly disagree. 
Although each item has four response options, the items are 
scored either positively or negatively.- Some of the items 
are combined so that scores range from 0 to 6 . 



5 to 10 minutes . 

About 1 minute per student, 
s tudents themselves . 



Items can also be scored by 



Grade levels for 
V7hich ins trument 
is appropriate: 

Technical quality: 



Relationship to 
program obj ectives : 

Use of subscales: 

Recommendation : 



Suggested alternatives; 



The scale V7as designed for juniors and seniors in 
high school • 



In his book Society and the Adolescent Self-image ,^ 
E.osenberg gave only a coefficient of reproducibility 
of .92 and scalability coefficient of .73. A small 
scale study by Seller and Tippett-- indicated that 
the test-retest correlation over tV7o vreeks was .85. They 
also found that the scale correlated from .56 to .83 with 
several similar measures and clinical assessments. 
Rosenberg presents a great deal of data about the relation- 
ship of the self-estcei'! scale with other measures - viz., 
neurosis, depressive effect, gloom and disappointment, etc. 

The self-esteem measure is an indicator of the students 
attitude toward himself, objective I-C. 

There are no subscales. 

The scale is both brief and thorough, which should benefit 
any. effort to assess attainment of the program objectives. 
It is highly reliable for such a short scale and could be 
used without grouping items as directed. Students could 
also administer the scale to themselves and score it 
themselves. Because the scale is short and well regarded by 
many social psychologists, it should be included in any 
attempt to measure student attitude toward themselves. 

A longer scale that may be worth consideration is Coopersmith' i 
'^Self-esteem Inventory** although Rosenberg's scale is better 
suited to an assessment program. 



-Rosenberg M. Society and the A dolescent Self-image . Princeton, N.J.: Princeton 
University Press, 1965. 

'■•-Silber, E L Tippett;, J. Self-esteem: Clinical assessment and measurement 
validation. Psychological Reports , 1965, 1^, 1017-1071. 
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Student inEormation siheet 



General descrip tion : 



This sheet is a two part questionnaire concerning the 
student's educational background. Ilie first part, in- 
cluding nine items completed by the student, asks about 
his sex, race , residence , at titude toward school , edu- 
cational plans, work experience, and thoughts about his 
future. The second part, completed by the student's 
teacher, asks about the occupation of head of household, 
the student's IQ> and percentile rank reading score. 



Administration time : 



Scoring time: 

Grade levels for 
which instrument 
is appropriate: 

Technical quality: 



Approximately 10 minutes for students and 5 minutes for 
each student by the teacher. 

Unknown 

A student would have to be 15 to 16 years old ±n order 
to answer questions 8 and 9 of Part I. 



Inhere is no information avaj.lable as to the reliability 
and validity of the questionnaire items. Based on previous 
experience with national survey data, it is likely that 
Part I, items 3, 4, and 6 would be very reliable, probably 
in the neighborhood of .60 to .70. On the other hand, items 
such as 7 and 9 are usually extremely unreliable - in the 
neighborhood of .10 to . 20 , which is unacceptable . Tlie 
quality of Part II, item 1, is questioned and may be 
invalid, for the most part. IQ score could also be sub- 
jected to some criticism depending on when the IQ score 
was obtained. 



Relationship to 
program objectives: 

Use of subscales : 

Recommendation : 



Sugges ted alternatives : 



None. 



All data is item data only. 

The student information sheet should not be retained in 
its present form. Background questions often seem 
appropriate in collecting data, for one reason or another. 
Nonetheless, there should be a specific reason for in- 
cluding every bit of information - i.e., all information 
collected should be used in the analysis of the outcome 
data in some fashion. If such variables are included in 
the analysis, then that is fine; otherwise, do not collect 
the data. Either expand the information sheet to improve 
the quality of the items or dr5p the potentially unreliable 
items. 

If such variables as attitude toward school (Part I> item 
7), future plans (Part I, item 9), and parents occupation 
(Part II, item 1-) are to be used in the analysis, these 
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constructs should be defined better and measured through 
scales rather than by single items. For example, an 
overall attitude toward school scale might contain items 
about counselors, teachers, courses, extracurricular 
activities, and difficulties with teachers and subjects. 
A ten item scale, perhaps called ''dissatisfaction with 
school," might look something like this: 



agree 
very 







disagree 


agree 


much 


1. 


Teachers always give me a hard time 


1 


2 


3 


2. 


I don't like the other students in this 










school very much 


1 


2 


3 


3. 


The school counselors don't help me 


1 


2 


3 


4. 


My courses are very boring 


1 


•2 


3 


5. 


I don't participate in extracurricular 










activities 


1 


2 


3 


6. 


My courses are too difficult for me 


1 


2 


3 


7. 


Teachers don't help me enough 


1 


2 


3- 


8. 


Other students at this school don't like 










me 


1 


2 


3 


9. 


I don't like the principal of this school 


1 


2 


3 


10. 


The other students in this school are not 










as smart as they think 


1 


2 


3 
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Measures Rev3.ewed for Possible Use 
in FY 76 



The instruments listed below were chosen for review based upon annotalied 
bibliographies of the ETS test collection. Additional instruments were 
rejected without review as 1) inappropriate for the age groups to be included 
in the evaluation of the Career Education Consortium, or 2) not measuring 
the project objectives. 

SeM-awareness ~ Grades 3»6 

"Self-esteem inventory," Stanley Coopersmith. In The Antecedents of Self- 
es teem by Coopersmith; W. H. Freeman and Co., San Francisco, 1967. 

Comment: No grade level norms; insufficient technical 
data, 

*'Self Report - Inferred Self-concept Scale" ("About Me"). James Parker, 
in "The Relationship of Self Report to Inferred Self-concept" by 
James Parker in Educational and Psychological Measurement, 1966, 26, 
291-700. 

Comment: No norms data, no reliability reported. 

"How I See Myself," Ira J. Gordon, Florida Educational Research and 
Development Council, University of Florida, Gainesville, 1968. 

Comment: Well documented, but response made is more difficult 
for young children than that used in the Piers-Harris 
scale . 

"Piers-Harris Children's Self-concept Scale" ("The Way I feel About Myself"). 
C. Piers and D. Harris, Counselor Recordings and Tests, Nashville, 
Tennessee 5 1969. 

Comment: Well documented, reviewed favorably in Buros* Mental 
Measurement Yearbook , 5th Edition. Format requires 
only a yes-no response. 

Career Knov7led ge - G rades 3, 6 

"Career Education Cognitive Questionnaires," B. Rader and K. Nelson, 

Minnesota Research Coorr^inating Unit for Vocational Education, University 
of Minnesota, Minneapolis, 1975. 

Comment: Field testing involved a relatively limited 
sample, but instruments are presently being 
used to evaluate a number of Part D projects, 
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and additional data should be available soon. 
Quality of printing on instruments, is not 
uniformly high'. Recommended by USOE Guidelines 
for Evaluation . 

^'Career Education Needs Assessment," A, Blome and G. Rask, Olympus 
Publishing Co., Salt Lake City, 1975, 

Comment: Good face validity in instrument for grades 
4-6, but no technical data was available. 
K-3 instrument requires individual admini- 
stration. 

Career Kn owled ge and Decision-Making - Grades 9, 12 

"Readiness for Vocational Planning" by Donald Super. In EmerRin^ Careers 
by Warren D» Gribbons and Paul R. Lohnes, Teacher's College Press. 
Columbia University, New York, 1968. 

Comment: Requires individual interviews, not appropriate 
for evaluation. 

"Career Development Inventory," Donald Super and David J. Forrest. Teacher 
College, Columbia University, New York, 1972. 

Comment: Not published; requires weighted scoring; in- 
cludes attitudinal and cognitive scales; 
data available from only one study of 400 
tenth grade students in Michigan. 

"Guidance Inquiry," M. Katz. ETS, Princeton, N. J. 

Comment: No longer available; redesigned as an 
instructional program. 

"Assessment of Career Development, Grades 8-11," American College Testing 
Program, Houghton Mifflin Company, Atlanta (Boston), 1974. 

Comment: Well documented, more comprehensive than 

others reviewed, good match of project objectives 
and test subscales; requires 125 irinutes of 
test time. 
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A Proposal for the Continuation of a Third-Party 
Evaluation of the Florida Comprehensive Program 
for Career Development: 
Kindergarten through Universities 



(The Third-Party Evaluation for the U. Commissioner's 
Discretionary Exemplary Vocational Project) 



A Technical Proposal 
to 

The Florida Department of Education 
by 

Educational Testing Service 
17 Executive Park Drive, N. E. 
Suite 100 
Atlanta, Georgia 30329 



This information and data furnished shall not be discussed outside of 
the Florida Department of Education and shall not be duplicated, used, 
or disclosed in whole or in part for any purpose other than to evaluate 
the proposal, provided that if a contract is awarded to this offeror as 
a result in connection with the submission of this information and data, 
the Florida Department of Education shall have the right to duplicate, 
use, or disclose the information or data to the extent provided in the 
contract. This restriction does not limit the Florida Department of 
Education's right to use information and data contained herein if it is 
obtained from another source without restriction* 
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!• Background 

In February, 1975, Educational Testing Service entered into an agreement with 

the Florida Department of Education, Division of Vocational, Technical, and 

Adult Education to plan and conduct an evaluation of the Florida Comprehensive 

Program of Vocational Education for Career Development as required by the 

Rules and Regulations of Seccion 142(c) of Part D, Vocational Educational 

Amendments of 1968, P. L. 90-576. In accord with that agreement, ETS has 

sought to 1) establish a v/orking relationship with the project management, staff, 

and participants, 2) review and organize available data concerning both the process 

and products of the project, 3) assemble information to support a certification 

that the program is, in fact, in operation, 4) identify processes and activities 

which are not on schedule in their development and thus endanger the success 

of the program, 5) solidfy sampling and data collection, procedures , and 6) develop 

the final evaluation design in congruence with the discovered realities of the 

program. 

Activities proposed by ETS for fiscal year 1975-76 build upon the activities 
and experiences of our previous involvement. Whereas most of the activities 
of the previous four months involved establishing relationships, orientation, 
and the development of an evaluation design, activities for the coming year can 
move to an implementation of the evaluation and more definitive feedback on 
project effects. 

As indicated in the original proposal of ETS, the specifics of evaluation 
design and the details of information collection will continue to evolve overtime, 
in step with the development of the project. The plan summarized in this proposal 
has already undergone several revisions and v7ill be revised again as the 
project changes. The plan presented, however, has been reviewed by the project 
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staff and approved as an appropriate approach to the evaluation of project 
activities as presently defined. 
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II. Product Evaluation 

The evaluation of the Florida Comprehensive Program of Vocational Education 
for Career Development can be considered in two parts: process evaluation 
and product evaluation. 

As generally defined, product evaluation is based upon the assessment of 
student changes related to behavioral objectives specified by the project. 
The Career Education Consortium, working cooperatively with the district 
career education projects in Manatee County and Sarasota County, have defined 
student objectives for elementary and middle school students in the second 
quarterly project, report, lliese objectives are classified according to the 
nine student outcome areas identified in Handbook for the Evaluation of Career 
Education Programs . It is anticipated that the consortium will, in the near 
future, identify additional student outcome objectives for secondary and post- 
secondary students. As additional objectives are defined, the product 
evaluation will be expanded to include the added dimensions. 

It is neither necessary nor feasible for the product evaluation to include 
a measure of every student outcome for every student. The project effects 
can be determined within limits of probability through a sampling of both 
outcomes and student participants through a controlled design. The principles 
for establishing such a design are outlined in the Handbook for the 
Evaluation of Career Education Programs . The proposed design follows these 
principles and will provide a reliable and valid estimate of the accomplishment 
of student objectives resulting from the implementation of career education 
activities in Manatee and Sarasota counties. 

Research Design : Student behaviors are influenced by a multitude of factors 
other than the project "treatment." To isolate the effect of the treatment, 

O 'j ^ 
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every effort must be made to '^control** other influencing variables. Ideally, 
extraneous variables can be controlled through the random selection of Ss and 
random assignment of Ss to a treatment or control condition. The random 
assignment of Ss is not possible in the Sarasota and Manatee districts, but 
limited control of variables associated with history, maturation, testing, 
instrumentation, and sample mortality can be obtained through a design 
utilizing a comparison group of students of the same grade placement who are 
experiencing none or very few of the career education services provided to target 
schools. In both Sarasota and Manatee, not all schools are equally involved 
in the career education project, therefore, a comparison group' is available. 

The design to be implemented is a pre-post comparison group design. The 
design is symbolically represented as follows: 

treatment group 0X0 

comparison group 0 0 

Sample Size : The evaluation budget will provide materials for testing 800 
students on each of two occasions. With tv^o districts involved, this is 400 
students per district. Three hundred students will be selected from the 
treatment group and 100 from the comparison group. 

Grade Levels and Variables to be Measured : A minimal program of student 
assessment is established by the previous contractual arrangement between ETS and 
and the Florida Department of Education: 

**In accomplishing this, the work during the current year shall include but 
not be limited to measures of the following: 

1) The extent to which students who have participated in the project 

demonstrate an increase in self-awareness in Grade levels 3, 6, and 9; 
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2) The extent to xdiich students who have participated in the project 
demonstrate an increased awareness of and knowledge about work at 
Grade levels 3, 6, 9, and 12; 

3) The extent to which students who have participated in the project 
demonstrate increased competency in career decision-making skills at 
Grade levels 9 and 12." 

This minimal program is outlined below: 



Objective 




Grade 


Level 






3_ 


6^ 


9 


12 


Self -awareness 


X 


X 


X 




Work-knowledge 


X 


X 


X 


X 


Career decision-making 






X 


X 


s amp le- tr ea tmen t 


150 


150 


150 


150 


sample-comparison 


50 


50 


50 


50 




200 


200 


200 


200 



Test Schedule : One week will be allowed for each test administration. The 
specific days and tiraes for testing will be coordinated by the local district 
Pretests V7ill be administered September 8-12, 1975, in Sarasota County and 
October 6-10, 1975, in Manatee County. Posttests will be administered 
May 17-21, 1976. This schedule allows a maximum treatment period without the 
interference of school starting and school ending events. 

Service Provided to the District by ETS : 

1) provide examination copies of instruments by August 15, 1975. 
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2) select sample from lists provided by district. 

3) conduct orientation workshop for person(s) responsible for testing at 
each school • 

4) summary of pretest results by January 1, 1976. 

5) suntmary of posttest results by August 1, 1976. 

6) all student materials, scoring, and analysis services will be provided 
by ETS. 

7) an ETS consultant v/ill be present on the days of and prior to the first 
administration in each district to ap,swer questions of procedure. 

Services' Provided by the District : 

1) confirm test dates. 

2) provide list of students in treatment groups, 
provide list of students in comparison groups. 

3) notify counselors > teachers, parents, etc. of test purposes, dates. 

4) provide list of persons to be responsible for administering tests at 
each school. 

5) receive, administer, and return all test materials*. 
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Data Analysis : Distributions and descriptive statistics of pretest scores 
and subscores for each grade level and treatment group V7ill be produced 
for each district separately as well as in combination. 

Posttest scores' for treatment and comparison groups will be compared using 
analysis of variance. However, if significant differences betv;een the 
treatment and comparison group pretes t . scores are obp Ived, analysis of 
covariance with pretest scores as covariant will be u^jj. to analyze 
posttest score differences. All analyses will be done for the two districts 
separately and for their combination. 

Role of USOE Guidelines for the Evaluation 
of Career Education Programs 

the^dpjaft guidelines prepared by Development Associates, Inc. for USOE 
are intended to aid career education programs in the evaluation process and 
are not intended as a hindrance to either program management or evaluation. 
The guidelines are li-mited in application to product outcomes for elementary 
through secondary students. The guidelines make no provision for process 
measures or measures of outcomes for nonstudent groups or pos tsecondary student 
groups. These are the two major areas of concern to the Career Education 
Consortium, 

Nevertheless, an attempt has been made to apply the basic elements of the 
federal guidelines model where appropriate to the consortium project. This 
effort v;ill be communicated to USOE and to Development Associates, Inc. in 
hopes that the guidelines will be both simplified and expanded to consider 
a wider range of outcomes. 
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Treatment Group-O u tcome Area Table 

The Treatment Group-Outcome Area Table serves several functions. This 
table identifies the various treatment groups. Each treatment group consists 
of project participants receiving the same services from the project. Each 
group> however, receives a different set of services. 

From a survey of project documents and from interviews with project staff, 
tv/enty-seven treatment groups are identified. These fall into three classes: 
students, faculty and staff, and the ^^usiness community. Table 1 lists the 
thirteen student treatment groups. The faculty and staffs of the student 
groups represent thirteen additional treatment groups. A twenty-seventh group 
is the business community. 

The TG-OA table also identifies as treatments those components listed in 
the consortium proposal or the proposals for the two district projects. 
Activities comprising the components are described in project documents. 

The third element of the TG-OA table is the classification of objectives or 
outcomes for each treatment group according to the nine major outcome areas 
identified in the Evaluation Guidelines . Tliesc are listed in Table 3. 

This r-^.sk demonstrated some of the fundamental weaknesses of the Guidelines ' 
design as well as some weakness in the planning documents for the consortium. 
The nine areas of outcomes listed in the guidelines all relate to student 
outcomes. Consequently > there is no system for classifying outcomes for the 
faculty-staff or business community objectives, and, therefore, many of the 
activities and goals of the consortium are not recognized by the Guidelines * 
design. 



The TG-OA table also points to a weakness in consortium planning. 

Goals and activities for s tudcats at the pos tsecondary level are not clearly 
stated. There is, at the present time, limited service available to 
pos tsecondary students. Most pos tsecondary activity has been aimed at faculty-- 
staff. As treatments reach the student level, project efforts should be 
turned to identifying treatment groups and expected outcomes. The same is 
true for private school groups and for handicapped students. 

Noting the two restrictions above, the TG-OA table can be appropriately 
completed only for the eight student treatment groups, elementary through 
secondary (this was obviously the intent of the Guidelines ' design). This 
should in no v;ay imply that the consortium should increase efforts in the 
elementary-secondary student programs to the detriment of other treatment 
groups. It is an artifact of the Guidelines ' design. 

The assignment of outcome areas to treatment groups is based upon the consortium 
analysis of project objectives presented as Appendix A of the Second Quarterly 
Report. 
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Table 1 

List of Student Trpatment Groups 

Students 

Elementary - K-3 

1. Manatee Co, 

2. Sarasota Co, 
Middle School - 4-6 

3. Manatee Co. 

4. Sarasota Co, 
Jr. High - 7-8 

5. Manatee Co. 

6. Sarasota Co • 
Secondary Schools 

7. Manatee Co- 

8. Sarasota Co. 
Jr. College 

9. Manatee Jr. College 
University 

10. University of South Florida 

11. Adult 

12. Private Schools 

13. Handicap 
Vo Tech 

14. Manatee Co. 
Vo Tech 

15. . Sarasota Co. * 
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Table 3 



Career Education Objectives in Terms of 
Student Outcome Areas 



I. Students will demonstrate increased self awareness. 

II. Students will demonstrate increased competency in basic academic/ 
vocational skills . 

III. Students will demonstrate increased awareness of work values and 
possess a desire to engage in paid and/or unpaid work, 

IV. Students will demonstrate increased awareness of and knowledge about 
work. 

V* Students will demonstrate increased competency in career decision- 
making skills • 

VI. Students will demonstrate good work habits. 

VII. Students will demonstrate work-seeking and work-getting skills. 

VIII. Students v/ho are leaving the formal education system will be success- 
ful in being placed in a paid occupation, in further education, or 
in unpaid work that is consistent with their current career education. 

IX. Students will be aware of means available for continued education once 
they have left the formal educational system. 



-13- 

OutcoiTie QuGstiog/Treatment Group Matrix 

The Outcome Question/Treatment Group Matrix Identifies the specific objectives 

of the Guidelines' design which are appropriate to each treatment group In 

the project. Since each of the Guidelines' objectives relate to student 

outcomes, this matrix Is completed only for student groups. The analysis 

Is based upon the comparison of consortium and county objectives to 

federal guideline objectives in the consortium's Second Quarterly Report • 

The outcomes for Manatee Junior College and University of South Florida students 

are not clear in project documents and have not been included. 
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Emphasis of the Evaluation 

Activities for the first year of the CEC project have centered on the 
orientation of staffs of the member institutions to the concept and goals 
of career education and to establishing lines of communication among the . 
member institutions » Target groups have been elementary and secondary teachers, 
guidance counselors., administrators, MJC and USE faculties, and the staffs 
of the district career education projects. Substantial efforts have also 
been devoted to building the knowledge and skills of the consortium staff 
members . 

During this phase, the evaluation efforts have included an accounting 

of contacts made and coiranunication lines established as well as the collection 

of baseline measures of student behaviors at the elementary and middle school 

levels. 

During the second year of the CEC project, contacts with elementary and middle 
school personnel will continue, but the project emphasis will shift to secondary 
and postsecondary levels where career education concepts and activities are 
not as fully developed or as well established. 

Evaluation efforts begun during the first year will continue with improvements 
based upon experience. Evaluation will also be expanded with attempts to go 
beyond an accounting of activities to systematically assess the impact of 
these activities on the target groups and programs of the project. 

The emphasis of the evaluation effort will be on "process.*' Many 
of the consortium activities do not involve direct student contact. 

1,U 
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The consortium is attempting to influence many nonstudent groups through 
workshops, committees, publications, and personal contacts. The logical 
expected consequence is that these nonstudent groups will undertake actions 
that will influence the education of students. The student effects resulting * 
from such an approach are often long range and may not be discernible during 
the early stages of the project. Evaluation to provide information for 
program management must, therefore, look for intertnediate indicators. The 
identification of intermediate indicators of project progress is a part of 
"Process Evaluation," 

One indication of the progress of CEC is a sample accounting of activities 
planned versus activities completed each quarter. The project quarterly 
report presently provides this accounting. A second level of Process 
Evaluation must focus on the impact of these activities on significant target 
groups. This impact is not presently being systematically assessed, and reported. 
During year two of the CEC project the third party evaluation will attempt to 
provide this systematic assessment of the impact of CEC activities on the 
knowledge, attitudes, resources, and actions of target groups whose actions 
are critical to the accomplishment of project goals. 

During the latter part of year two and on into year three, career education 
programs for MJC and USE students should develop to a stage where 
it is reasonable to expect some change in student outcomes. The evaluation 
plan V7ill develop V7ith these programs such that student outcomes at the 
postsecondary level will be assessed when programs are implemented. 

For the immediate future, the greatest evaluation need of the Ct^C prujc^:: 
is the area of Process Evaluation. Product evaluation at elementary and 

1 i. r-^i 
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secondary levels is underaay; product evaluation at pos tsecondary levels 
would, at this point, be premature. A majority of CEC activities do not 
involve direct contact with students. These activities are nevertheless 
critical and should be evaluated. No guidelines, models, or designs 
for Process Evaluation have been provided to the project. For these reasons, 
a major emphasis of this planning document is the presentation of a conceptual 
model for the "Process" elements of the CEC project. 
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III. Process Evaluation . 

Product evaluation is based upon the outcomes of a project in terms of 
behavior changes. In contrast, process evaluation attempts to describe and 
assess the effectiveness of activities , components , and transactions which 
are intended to produce the outcome effects. Tlie steps to be followed in 
the process evaluation for the CEC project include 1) definition of a sampling 
base representative of all project activities, 2) selection of a sample of 
elements from the defined base, 3) a listing of process objectives and proposed 
activities from project documents for each element selected for evaluation, 
4) preparation of evaluation questions and sources of information for each 
process objective or activity identified, 5) collection of information for 
each evaluation question from project documents, staff, and participants, 
6) analysis of data> and 7) preparation of a descriptive and evaluative 
report of findings and conclusions. 

1) D€! finition of sampli ng base: The design for process evaluation of the CEC 
project begins v/ith the description of the project structure, components, and 
activities provided by the project planners. A three dimensional representation 
of the project model is presented as Appendix N of the project Third Quarterly 
Report and is reproduced as Figure 1 of this proposal. The dimensions of this 
model include 1) student levels, 2) service components, and 3) management 
functions (labled "activities" in Figure 1) . 

Figure 2 is a matrix combining the elements of the structural model for the CEC 
project. This matrix V7ill form a basis for sampling from the many activities 
of the CEC project for the purpose of evaluation. This procedure is somev7hat 
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Figure 1 



Conceptual Model for CEC Project 




analogous to sampling behavioral objectives for the purpose of product 
evaluation. The elements of the matrix represent the population of all 
activities undertaken by the Florida Career Education Consortium. It is not 
possible to observe and evaluate all project activities, therefore, a systematic 
sample of activities is drawn as representative of all activities. 

2) Selection of Sample ; The elements of the CEC project structure to be 
evaluated during the 1975-76' fiscal year and the quarter in which each element 
will be evaluated are indicated in Figure 2. These elements have been selected 
by the evaluator to represent a broad spectrum of project activities. Not all 
elements of Figure 2 are receiving equal emphasis in the CEC project. The 
process evaluation will attempt to describe and value the status. of the 
elements selected in relationship to project emphasis, goals, and plans and 
will include recommendations for changes in direction or effort. 
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3) Identification of process obje ctives and activities ; The process evaluation 
for each element selected will be structured according to the five management 
functions listed for each component in Figure 2 (Planning, Development, In-service ^ 
Evaluation, and Articulation). For each management function, the evaluator will 
identify relevant process objectives, tasks, and activities specified in CEC 
project quarterly reports. As an example, some of the activities planned for 

the curriculum component at the secondary level are listed in Table 1 of the 
Appendix. (This component is scheduled for evaluation the first quarter of 
FY 76). For each planned activity, ETS will seek to identify evidence that the 
activity has been accomplished or is proceeding according to plan. Of course, 
not all activities of the project can be anticipated and included in the project 
quarterly reports • Every effort will be made to identify unanticipated activities 
and outcomes vzhich relate to the specific component and level under investigation 
and to include these in the evaluation. 

4) Evaluation questions and sojjrces : For each activity or process objective 
identified in step 3, the evaluator will specify a related evaluation question 
and identify the source most capable of providing the required information. The 
primary sources will include CEC staff members, members of CEC committees and 
task forces, staffs of the two district projects, project documents, such as 
meeting minutes and newletters, and participants of in-service activities. 
Evaluation questions and information sources for the curriculum component at 
the secondary level in Sarasota County are included in Table 2 of the Appendix 
as an example. 

5) Collect ion of data : Data collection will be planned such that required 
interviev7S for the evaluation of each project element can be completed during 
one site visit. The schedule of site visits for the collection of process 

118 
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evaluation is included in the Evaluation Calendar, p. 37. Wliere the evaluation 
requires a review of project documents, the documents will be collected during 
the site visit and returned within thirty days. 

6) Analysis of data by managem e nt function : Although each activity or process 
objective will be individually evaluated, an attempt will also be made to draw 
some generalizations concerning groups of activities which relate to specific 
management functions. The functions to be analyzed include planning, three 
subdivisions of implementation (development, in-service training, and articu- 
lation), and evaluation. Table 1 in the Appendix groups activities for the 
curriculum component at the secondary level in Sarasota County by management 
function as an example. 

For each relevant management function within a project component and 
level being evaluted, a quality judgment will be made. 

In relation to the process objectives, project accomplishments will be evaluated 
with respect to resource utilization, effectiveness, documentation and products, 
and problem identification and resolution. 

In the area of Resource Utilization the evaluator will provide a subjective 
judgment of effective utilization of time, money, and personnel in relation to 
components and activities and to achievement of process objectives. 

In the area of Effectiveness the evaluator vrill provide a value judgment on 
a four point scale (excellent, good, fair, poor) of the effectiveness of each 
activity in accomplishing its purpose. Separate ratings will be provided based 
upon each of three sources: 1) CEC staff perceptions, 2) Activity participants' 
perceptions, and 3) Objective data. Objective data might include cognitive 
measures administered to workshop participants, counts of requests for materials 
and consultants, descriptions of new programs, etc. iXij. 
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In the area of Documentatio n and Products the evaluator will provide a 
rating of the completeness and overall quality of 1) any report or other 
documentation outlining what was done (activity report), 2) any report of 
the end result of the task or activity (product report), and 3) any recom- 
mendations concerning the activity and its application to a "model" program 
(recommendations) . 

In the area of Problem Resolution , the evaluator will provide a rating of 

the project's attempts to 1) identify problems, and 2) take corrective actions. 

l^erever possible, criteria and sources for the evaluator' s judgments will 
be documented or referenced. 

A summary chart such as that in in Figure 3 will be presented for each element 
of the project evaluated in addition to a narrative report for the evaluation 
of specific activities. 
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Figure 3 

Process Evaluation Summary by 
Management Function 



Curriculum Component 
Secondary Level-Sarasota 


Resource 
Utilization 


'Effectiveness 


Products and 
Documentation 


Problem 
Resolution 


Function 


Schedule 


Manpower effort 1 

1 


' Expenditures 


Staff perception 


■'1 

Participant perception 


Objective data 


Activity report 


Project report 


Recommendations 


Identification 


Strategy 


1 - Planning 
























2 ~ Implementation 
























a. Development 
























b. In-service 
























c. Articulation 
























3 - Evaluation 

























^Entries will be on a four point scale from excellent to poor based upon a synthesis 
of activities investigated for each function. 



7) Preparation of reports: The report for each component and level evaluated 
will be presented in two parts. The first part will identify activities 
investigated, methods of investigation and a description of the status of the 
activity based upon the information gathered. For example, if the process 
objective "encourage joint staff meetings'* is investigated, the descriptive 
report will include the number of such meetings held, the topics discussed, 
and the evaluative comments of those participating ► 

The second part of the report will present some professional judgments of 
the evaluation team based upon the evidence accumulated concerning each activity 
and for each set of activities comprising a management function for the component 
and level under investigation. The criteria specified in step 6 will be applied 
to each activity and summarized by management function. 

The evaluation of the "process" elements of the CEC project clearly should 
be approached V7ith an understanding of the project philosophy and priorities. 
Much time has been spent during the previous six months orienting Dr. Hardy 
and other ETS staff members to this task. The vantage point of a third party 
evaluator is uniquely different from that of the project staff or members of 
FDOE. The plan outlined in this proposal will provide information and view- 
points not available to the project management from other sources. 

Services provided to the project by ETS : 

1) Provide plan of activities to be investigated and dates for site 
visits (included in this document). 

2) Provide list of evluation questions and persons to be interviewed for 
each site visit at least one week in advance. 

3) Provide a trip report to the project director describing activities 
and findings. 

4) Provide oral and written reports to include descriptive and judgmental 

data on components examined to date each quarter, 

1 ^ 



-36- 



Services provided by the Project : 

1) Confirm site visit dates as appropriate. 

2) Schedule required interviews with project staff and participants. 

3) Provide copies of all project documentation and reports. 

4) Inform evaluator of upcoming significant project events. 

5) Review and approve or amend all evaluation reports. 
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IV» Evaluation Calendar 

Schedule of On Site Visits 

Date Purposes 

Sept. 4, 5, 8, 9 1. Review test administration. 

procedures with Sarasota County 

2. Conduct interviews for evaluation 
of Curriculum component at 
Secondary Level in Sarasota County 

Oct. 3, 6 1. Review test administration 

procedures with Manatee County 

Oct. 20-23 1. Conduct interview for evaluation 

of Placement and Follow-up at MJC 

2. Conduct interviews for evaluation 
of Guidance component at MJC 

Jan. lA-16, 1976 1. Evaluation of Guidance component 

in Private Schools 

2. Evaluation of Placement and Follcw-up 
component at the community level 

Apr. 14-16 1. Evaluation of Guidance component at 

Elementary Level in Manatee County 

May 13, 14, 17 1. Review posttest procedures with 

Manatee and Sarasota School districts, 
monitor testing. 
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Report Dates 



Date Title 
Oct. 10, 1975 1st quarterly report 



Jan, 9, 1976 2nd quarterly report 



April 9, 1976 3rd quarterly report 



June 30, 1976 4th quarterly report 

Final Technical Report 



Content 

a. Elementary and Secondary 
product pretest data from 
Sarasota County. 

b. Process evaluation of 
curriculum component at 
secondary level in Sarasota 
County. 

a. Elementary and secondary 
product pretest data from 
Manatee County. 

b. Placement and Follow-up 
at MJC. (Process) 

c. Guidance component at MJC. 
(Process) 

d. Postsecondary product 
evaluation design. 

a. Process evaluation of 
guidance component in 
private schools. 

b. Placement and Follow-up 
at community level. 

c. Postsecondary pretesL produ< 
measures (if project goals 
at postsecondary level are 
defined) . 

a. Process evaluation of 
guidance component at ele- 
mentary level in Manatee 

. County. 

b. Posttest data for elementar 
and secondary levels in 
Manatee and Sarasota Counti 

c. Posttest data for post- 
secondary level. 

d. Summary of product and 
process evaluation findings 
conclusions, and recom- 
mendations. 
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Table 1 



Selected Activities Planned for the 
Curriculum component at the Secondar y 
level in Sarasota County 



Activity 



PLAtTNING 



1» Reviev/ career education plans 
and programs in the district 
relative to 

1,1 occupational awareness 

1»2 work experience 

1.3 employability skills 

1.4 job entry skills 

2» Encourage joint staff meetings 
between district staff and 
consortium staff 

3. Gather data regarding assess- 
ment of present programs in the 
district 

4. Identify methods for improving 
occupational education programs 



5. Determine need for work experience 
programs within service area 



5. Assist in identifying CE 

consultants for district teachers 



Source 



Objectives 1-4 
pp, 64-90 

1st quarter report 



Objectives 1-4 
pp. 64-90 

1st quarter report 

Objectives 1-4 
pp. 64-90 

1st quarter report 

Objectives 1-4 
pp. 64-90 

1st quarter report 

Objectives 1-4 

pp. 64-90 ' 

1st quarter report 

Appendix I, p. 108 
1st quarter report 



DEVELOPMENT 

1. Compile lists of materials, 
consultants, and other information 
by subject area 

2. Development and collection of 
materials relative to broad 
objectives 
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(p. 16, 3rd quarter, report) 



(p. 11, 2nd quarter report) 
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Table 1 (Continued) 



Activity 

IN-SERVICE 

1, Conduct orientation workshops 

2. Workshop for secondary department 
chairmen 

AR TICULATION 

1, Select Articulation Advisory 
Committee 

2, Establish system of communication 
(nev7sletters , field visits, etc,) 

3, Identify deterrents to articulation 

4, Refer deterrents to subcommittee 
for resolution, 

EVALUATION 

1. Evaluation of existing district 
programs 

2. Evaluation of in-service training 

3. Evaluation of materials and methods 



Source 

(p. 106, 1st quarter report) 

(p. 16, 3rd quarter report) 

(p. 67, 1st quarter report) 

(p. 67, 1st quarter report) 

(p, 67, 1st quarter report) 

(p. 67, 1st quarter report) 

(p, 65, 1st quarter report) 

(p. 109, 1st quarter report) 

(p. 9-10, 2nd quarter report) 



In reviewing the list of planned activities in Table 1, information is required 
from at least four sources to evaluate the extent and effectiveness of these 
activities. A principal source for the evaluation of all activities is the CEC 
staff with particular emphasis on the project manager, Dr. Selman, and the 
persons responsible for secondary curriculum, Dr. Wu and Dr. Melton. Since 
there must be a close' relationship between the CEC project and the district CE 
project, the evaluation also includes questions for the district project director 
and/or the member of his staff most responsible for secondary programs. 

A third source of evaluation information for the curriculum component at the 
secondary level in Sarasota County is secondary level teachers. Teachers are 
the intended recipients or'iiargets for several CEC activities. The extent 
to which teachers have been effected can best be assessed by direct communication 
with some of these teachers. ETS will request 20 minute interviews with at 
least five teachers in Sarasota County. 

The fourth and final source of evaluation information is project documentation. 
Several planned activities imply a document as a final or interim product 
(lists of materials and consultants, evaluation reports, meeting minutes, etc.) 
ETS will request and reference copies of documents alluded to in project plans 
and v/ill ask the project staff to identify any additional relevant documentation. 

Table 2 lists some questions to be included in interviews for evaluating process 
aspects of the curriculum component at the secondary level in Sarasota County. 
Each question is related to one of the activities listed in Table 1. 



Table 2 



Process Evaluation Questions and Sources for the Curriculum 
Component at the Secondary Level in Sarasota County 



Sources 









:aff 1 










Staff 


District St 


Teachers 


jments 


Ques tions 


o 
w 


o 
o 


Planning 










1. 


\^at are the major career education activities and plans for 
Sarasota County? 










2. 


Have joint staff meetings included secondary curriculum topics 
and problems? 










3. 


Has assessment data been gathered and synthesized for Sarasota 
County at secondary level? 


s/ 








4. 


VJhat program changes would result in an improved CE program at 
the secondary level in Sarasota County? \7hat evidence indicates 
these changes would be an improvement? 


I' 




/ 




5. 


Has a survey of work experience programs and opportunities in 
Sarasota County been conducted? \n\at work experience programs are 
available? 










6. 


Have lists of consultants for classroom teachers been prepared? 
Are teachers aware that this resource is available? Has 
assistance in identifying CE consultants been requested by any 
Sarasota County secondary level teacher? Has this assistance been 
provided? 






;^ 




Development 










1. 


\NTiat materials and consultants are available to help a Sarasota 
County teacher integrate career education into his (math, history, 
etc.) class? 






/ 


/ 


2. 


\sTiat materials related to self awareness (or career knowledge or 
decision-making) at the secondary level have been collected or 
developed by the CEC project? 

1 3 0 


/ 









Table 2 (Continued) 



Process Evaluation Questions and Sources for the Curriculum 
Component at the Secondary Level in Sarasota County 



Sources 



Ques tions 



CO 
O 

u 

W 



W 

u 

0 

<u 
B 

o 
o 



In-Service 

1. \niat proportion of the secondary level administrators and teachers 

have attended a career education orientation meeting or workshop? 

2. Would you tell me briefly and in your own words, what is Career 
Education? 

Evaluation 

\^at evaluation of the program at the secondary level in Sarasota 
County has been conducted? \^at have been the most significant 
findings? How would you evaluate the Career Education Project 
in Sarasota County? 

\^at evaluation of in-service training for secondary level staff 
has been conducted? ^^^lat were general findings? Who uses the 
evaluation findings and for what purpose? 

\^at materials and/or methods have been evaluated, are under 
evaluation or will be evaluated for use at the secondary level 
in Sarasota County? 

Articulation 

Is there an articulation advisory committee? How often have 
they met? What issues have been discussed? What articulation 
problems have been solved or partially solved? 

\^at methods are used for communication betv/een the project and 
^secondary staff in Sarasota County? UTiat contact have you had 
with the Florida Career Education Consortium? 

\^at specific articulation problems have been identified that 
directly effect secondary level students in Sarasota County? 

. Has a committee been assigned the task of resolving any identi- 
fied deterrents to articulation? 



i/ 
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